Installing Git and GitLab on a Linux Server

Installing Git and GitLab on a Linux Server

As I said in my previous post, I am building a development environment. The starting point with any development environment is source control. I am going to use Git with GitLab as a UI over it. I’ll be installing it on a Ubuntu server.

First Steps: The Git basic setup

First, get your hands on a Linux server. Once you have it, log in as an unprivileged user and set up Git (taken from here and here), using the following commands:

Update system and install git:
sudo apt update (update apt cache)
sudo apt upgrade (upgrade to latest packages)
sudo apt install git (likely already installed)
sudo reboot (for good measure)

Now we set up the git user for the server
which git-shell (this will tell you the path to the Git shell)
sudo vi /etc/shells

add /usr/bin/git-shell (or the output of the which command) to it.

Now add a user for git to run as:
sudo adduser --disabled-password git
when prompted, leave fields blank

Next we set up the Git user – this user is used to commit files and will not have a shell or password, and will use authorized keys to send work to the server. We won’t be using it, however, since we’ll use GitLab, which communicates over http.

sudo su git
cd
mkdir ~/.ssh && chmod 700 ~/.ssh
touch ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys
vi ~/.ssh/authorized_keys

Copy your public key into the authorized_keys file. Again, not needed for GitLab, but needed to use git directly, as I mentioned above.
Then exit the git user’s shell and lock them out by setting their shell to the git shell. Since this user has neither a password not a shell, they can’t log in directly, even with a key.

exit
sudo chsh git -s $(which git-shell) 

Once that is done, you can test whether git works:

cd /home/git
sudo mkdir test_app_1.git
cd test_app_1.git/
sudo chown -R git.git test_app_1.git/
sudo git init --bare && cd ..

This should output a successful git setup.

GitLab setup

Next, we will install GitLab (instructions copied from here)

Firs, install dependencies:

sudo apt install ca-certificates curl openssh-server postfix

(for the postfix setup – I think internet site is what you want (it will want to send messages out), which we may set up later. For now, we won’t be setting up email integration, so any choice is fine.

Now, navigate to the temp folder, install the repository for GitLab, and then install GitLab itself:

cd /tmp
sudo curl -LO https://packages.gitlab.com/install/repositories/gitlab/gitlab-ce/script.deb.sh
sudo bash /tmp/script.deb.sh
sudo apt install gitlab-ce

Then configure GitLab by editing its config file and setting the URL – that’s all you need.

sudo vi /etc/gitlab/gitlab.rb

Set external_url to the external DNS name you set up for your host with your hosting provider. You’ll want to use the hosts file editor to set the internal IP address mappings on servers on your dev network that need to access it. So, if your Git/GitLab server is internally 10.0.0.8, you will need a hosts entry that contains something like:

10.0.0.8 gitserver.eastus.cloudapp.azure.com

I didn’t use https since it’s not worth the trouble in this situation, since LetsEncrypt requires certain gymnastics to get to work, and since this system will not be on all the time, it won’t be able to get new certificates.

Once you have set the external_url, run:

sudo gitlab-ctl reconfigure

Now, you can chill for a while while it sets up GitLab.

After it was done I rebooted once more for good measure.

At this point, you have a git server you can reach on port 22, but no means of accessing your GitLab instance. This is where the dev VM comes in. Since we’re cutting corners on security, we’ll only accessing the GitLab ui and data over http locally in the cloud network. I would not recommend opening port 80 to this server.

So spin up a VM (where you’ll be doing your work). You’ll want  the following software installed:

  • Visual Studio Community Edition
  • Hosts File Editor
  • Git for Windows
  • The GitLab Extension for Visual Studio (Optional if you want to experiment, but I don’t recommend it)

On your Dev computer, use Host File Editor to give an alias to the Git machine using the domain name you specified in gitlab.rb:

gitserver.eastus.cloudapp.azure.com maps to 10.0.0.8

Open a browser and navigate to: http://gitserver.eastus.cloudapp.azure.com

GitLab First Screen
GitLab First Screen

And you’ll get a GitLab page! So use your password manager to generate a secure password, and enter it and you’re up and running!

Now you’re ready to rock. Log in with username ‘root’ and the password you created, and click the  wrench in the toolbar to open the admin area. Click Users in the left menu, then click the green new user button and create your user account (you probably don’t want to be using the root user).

Now, create your new user. Since we didn’t set up smtp mail, you’ll want to set a first password. Then, log in as that user and change the password (again using a password manager to generate a strong password). Now you’re ready to check in code!

Create a user in GitLab
Create a user in GitLab

Checking in some code (the first, bad way)

Open visual studio and create a project, any project.

Just a basic Console Project
Just a basic Console Project

Right-click the solution and click ‘Add Solution to Source Control’:

Add solution to source control
Add solution to source control

This will create your local Git Repository. You will see that it worked in the Output pane under: Source Control -Git:

Source Control Panel
Source Control Panel

Now you can experiment with changes and local commits. Once you are happy with the commits, it’s time to push this to your GitLab server.

In order to sync with GitLab, the first thing I tried was the GitLab Extension.

The problem is that the ‘Sync’ pane has a ‘Publish to GitLab’ area that made me think that it could only publish to https://gitlab.com/:

The Misleading GitLab Publish Prompt
The Misleading GitLab Publish Prompt

The first time you try to commit and sync:

Commit and Sync
Commit and Sync

It will get upset and show this:

A Very Unintuitive Message
A Very Unintuitive Message

So click ‘publish’.

Publish To GitLab
Publish To GitLab

The first time you do it it will prompt for a server and credentials, then it will push to GitLab and set up the project:

Project is Now In GitLab
Project is Now In GitLab

As you can see, this worked. However, I can’t for the life of me figure out how to change the server or password, but it’s a start.

It turns out that this extension is sort of dubious, so I went back to this and tried to figure out a better, more sensible way to do this.

Using Git Native Mode With GitLab

So here is what works better. GitLab can also access its repository over http but appending ‘.git’ to the url of your project. To use this approach, create your solution and add to source control, as above.

Using your Test User, Create a project with the same name as your Solution (or another, it doesn’t really matter).

GitLab Project Creation
GitLab Project Creation

Copy the Project URL to the clipboard:

http://bghgitexample.eastus.cloudapp.azure.com/testuser/testprojecteta2

In VS, go to the Team Explorer, Click ‘Home’ (the house) and choose Settings, then ‘Repository Settings’. Then, in the ‘Remotes’ section, click ‘Add’.

In the Dialog Box that appears, past in the Project URL, and append ‘.git’ to it.

http://bghgitexample.eastus.cloudapp.azure.com/testuser/testprojecteta2.git

Git Remotes
Git Remotes

Now, in the lower pane, switch to the Package Manager Console (we’re using it just for the convenient PowerShell Console) and enter:

git push --set-upstream TestProjectEta master

This will return a weird that only occurs because we’re using the PowerShell in VS (you can run it in the developer powershell to not show it) but it will then connect your local repository to the project repository .

Package Manager Console
Package Manager Console

Now you can go to Team Explorer -> Home -> Sync and all the Actions are working:

Git Working with GitLab
Git Working with GitLab

The method overcomes the weirdness of the GitLab Extension and gives you a bit of the best of both worlds, it seems.

What I’m listening to as I do this: MC Frontalot’s Nerdcore Rising. A classic of the Nerdcore genre, he’s got some great songs on here. I’ll give Weird Al credit for really the first Nerdcore songs (All about the Pentiums), it really came into being in the early aughts with MC Chris, Optimus Rhyme, and Frontalot.

Building a Small Development Environment with Git, GitLab, Jenkins, and MSBuild

Building a Small Development Environment with Git, GitLab, Jenkins, and MSBuild

It’s been a little while since I posted about a tech project – I’ve spent the first half of the year exploring the Dark Souls universe and watching a bunch of Anime. But now I have a project to do and while researching how to do that project, I found that there doesn’t seem to be a resource on the internet for it.

The Project

My wife is starting a non-profit that is going to need a software platform, and I volunteered to help. However, it’s been a long time since I started something from scratch without any infrastructure, and as I searched the internet, as usual, I found lots of parts of a solution, but not an end-to-end guide to getting a workable build environment built. So I’ll share my notes here in case they happen to be useful to anyone.

My Skills and the Core Tech I’ll Use

I have worked the majority of my career in the Microsoft world, and the project will need a database, so I’m going to center this around a MS SQL Server database, with a C# middle layer/BFF using Entity Framework, and Angular as a front end.

The goal is to have a Development, Test, and Production build system that effectively means that no human interaction is needed with the servers.

I’m going to host this in Azure in a mix of VMs, and I’m going to try to balance security with expediency, with a first goal of getting the whole system working, and the second of gradually hardening the solution. The main issue here is that large scale corporate programming does have a lot more resources, so I want a low-cost reliable system that I can store in the cloud but won’t be a huge burden to manage.

The Software

For this task, I’m going the use the following software.

Git: It’s free, ubiquitous, and a platform I need more experience in. I have been in the TFS (now Azure DevOps Server) world for years, and so I need to get my arms around Git.

GitLab: I wanted a UI for Git for a few reasons. Partly due to my unfamiliarity, and partly because if gives a lot of features out of the box. It also has CI/CD and a ton of other features.

Jenkins: I’m not, actually, going to use the CI/CD pipeline from GitLab, though. I’m going to use Jenkins. I know it better, and it’s great software.

Visual Studio Community Edition: Yes, I could use VS Code, but again, I know VS really well, especially its source control integration.

The Architecture

Basic Lab Setup
Basic Lab Setup

I have an MSDN account, so I’ll use my Azure credits there to flesh this out. I’m going to use 4 VMS to start with:

  • Git server: An Ubuntu server running Git and GitLab. It will have 2 vcpus and 8GB ram.
  • Jenkins Server: a Windows 10 VM running Jenkins. It will have 2 vCPUs and 8GB RAM. It’s on windows so I can run MSbuild and MSDeploy. These might work on Linux, but I see no reason to find out right now.
  • Development server: A windows 10 server with Visual Studio and SQL Server Express. I’ll ultimately move the SQL server to a real box, but SQL instances are expensive, and for now I can live with SQL express. It has  4vCPUs and 32GB RAM.
  • IIS Server: I may or may not end up using IIS (maybe I’ll do an asp.net self-hosted app), but for now, IIS is a pretty good solution.

I’m not doing AD or any sort of domain name service, I’ll be just editing hosts files for this and using internal IPs. For this scale, it’s a bit of a pain to set up, but internal DNS isn’t worth the hassle.

Security to start will be pretty weak, but the goal is to get the environment built and working.

Other Software

In addition to the above, here are other bits of software you’ll need:

  • Host File Editor. It’s for editing host files. It’s one of those great pieces of software you didn’t think you needed until you found it.
  • Google Chrome: I may be using windows, but I have no reason to use Edge.

Next Steps

In the next few posts, I’ll go through the construction and configuration of this environment. The rough steps will be:

  1. Install and configure Git, Gitlab, and a dev machine.
  2. Install and configure Jenkins.
  3. Build the software and the deployer jobs.

What I’m listening to as I do this: DragonForce’s new album ‘Extreme Power Metal‘. DragonForce does one thing, and it does it well. It goes fast. Also, they covered ‘My Heart Will Go On’ on this album. This version of that song should have been the first dance at my wedding (not a household consensus, though).

 

 

 

 

Lean and the Team

In my last post, I talked about comparative advantage and how it can aid you in developing your team and your members’ skills. Now, I’ll talk about another aspect that you need to consider in a lesson from Lean.

I have never described myself as a person who espouses any one single methodology for solving problems, so you won’t ever see me write about the one right way of doing things. Many processes and frameworks were originally designed for one thing, and I have had uneven results applying them to software development. One thing that I have found to be very helpful in developing a team is a specific subset of the Lean concept of muda, which I came to understand when reading the book The Goal.

The Goal, Eliyahu M. Goldratt
The Goal, by Eliyahu M. Goldratt

If you haven’t read this book, it’s highly regarded and a bestseller for making certain business concepts very accessible to the reader. I highly recommend it.

The book has two great takeaways – first is the answer to the question: What is the goal of a company? The answer: To Make Money. It’s a very handy thing to keep in mind.

The second takeaway is the Theory of Constraints. This is the idea that production bottlenecks can take an assembly line and slow it to the speed of its slowest machine. Continue reading “Lean and the Team”

The Small Team

In my last post, I wrote about the choice between specialists and generalists on a team, using the metaphor of role-playing games to illustrate it. Now, I’m going to write a bit, over the next few posts, about my practical experience building and growing teams. This isn’t meant to be a guidebook on team development, or anything, but more of my approach and philosophy on team building.

D&D 1st Ed rulebook.
It’s all been downhill since then…

Sure, I also happen to have another awesome picture for this post that I feel reinforces my credentials in Role Playing Games, but that’s a total coincidence, I swear. Yes, I do just have these books lying around my office.

 

At many jobs I’ve had leading teams, I’ve started with somewhat small teams. For illustration purposes, the team I’ll describe will be an example, and not really any particular team I’ve actually had. Continue reading “The Small Team”

Risk, the Developer, and Patton

Every developer has been confronted with the request, ‘Take your best guess,’ in working their way through a problem. Now, this puts an engineer in a very hard position. Many teams are conflicted when it comes to taking chances: you can take any chance you want, as long as you only pick the right solution. That makes it very hard to put a developer in that position, since it’s not really a guess if the only allowed outcome is 100% success. Continue reading “Risk, the Developer, and Patton”

Books about Software Development: Life the Universe and Everything

Most readers will be intimately familiar with Douglas Adams’ work. If you’re not, it’s definitely worth a read. One item that relates to engineering and project planning that always springs to mind too late in a project was a concept he included in Life, The Universe and Everything: The ‘SEP’ – ‘Somebody Else’s Problem’.

This concept is basically a sort of cloaking field where people ignore something by assuming it’s somebody else’s problem.

“An S.E.P., […] is something that we can’t see, or don’t see, or our brain doesn’t let us see, because we think that it’s somebody else’s problem. […] The brain just edits it out; it’s like a blind spot.”

Douglas Adams. Life, The Universe and Everything, Chapter 3 Continue reading “Books about Software Development: Life the Universe and Everything”

Four Quadrants

A while ago I was in my car, listening to the radio, and the Beatles came on. I’ve never really been into them, but I have respect for them and their influence on music. I started to think about my likes and dislikes in four quadrants as sort of a silly thought experiment:

Four Quadrants
Four Quadrants

On the Y axis is my level of respect for something, and on the X is how much I like something. Continue reading “Four Quadrants”

Fear and Software

Several years ago, as a consultant, I had the opportunity to work with a client on modernizing their Quality Assurance processes. At the time, I was at the phase of my career where I viewed QA as just ‘testers’. They were sort of secondary characters in a play that starred programmers. Like the ponies in the Lord of the Rings. Important, sure, but they didn’t get their own swords or anything.

So I took a deep dive in to Quality and started at (close to) the beginning, with W.E. Demming. He was one of the first Quality gurus, and became known for his work in Japan following WWII (he has a wikipedia page). The one thing that really resonated with me from researching him was his idea that you had to drive out fear from engineering. At the time, I was still in the ‘bugs are bad’ phase, and because of that, made all the classic mistakes in programming. I wasn’t good at testing my own work, and wasn’t particularly good at telling our testers where to look for bugs. I also viewed testers more as impediments to my amazing software being released than as allies. Continue reading “Fear and Software”

Possible But Not Practical

I recently was working on a project and got in one of those discussions where I was not communicating at all. I was trying simultaneously arguing that the software was something I could build, but one that would not work. I felt pretty confident in my abilities to build and deliver it, and, in fact, that it would be pretty cool once built.

However, I was also trying to indicate that because of its complexity, it would have the characteristics of an unstable nuclear isotope – it would not last long. The system in question wold require so much more human intervention to keep it correct than was available within the organization that every month of operation would put it further and further out of true. It was ‘Possible, but not Practical’. Continue reading “Possible But Not Practical”