GIT Tip of the day

 

I have been using GIT for a while now on my Windows 7 Box,IMac machine and my Ubuntu machine , and have noticed sometimes that you get some issues with different character sets used on the difference platforms.

I had a issue again with a remote repository this morning by receiving the following error when trying to push my source back up to my remote repo :

Warning: LF will be replaced by CRLF in FILENAME

In order to fix this you can run the following command in your git shell on windows or normal shell on Mac and Linux :

git config core.autocrlf false

This will tell git not to try and convert the line-feed characters every time you try and push your source.

On another note , if you are new to GIT or looking for a good open source version control system. I would suggest you look at GIT as a option for your version control system. You can also look at alternatives such as mercurial or the good old svn (sub version) , that I have also found to work quite well for me.(but my personal preference is GIT) I have found that GIT works really good on Mac and Linux, and with  a little bit of effort , it works just as easy on windows, but be warned that the git shell for windows makes use of the Linux command system (Bash shell)

 

Here is some helpful stuff to get you started on git on the different platforms:

WindowsMac OSXSource

Multiple Platforms

  1. Git user manual
  2. Smart Git Multi-platform client
  3. meld is also good as far us I understand, but haven’t given it a bash yet.

Windows 

Windows Only

  1. Tortoise GIT client for windows
  2. msysgit tools for windows (this is what you need to get going on windows)
  3. BONOBO git server – if your maybe looking at hosting your own repo locally

Mac OSX

Mac Only

  1. Tower client (MAC only) for git (not free but excellent)

Source

Linux Only (Tested on Ubuntu – “WORKS ON MY MACHINE Winking smile“)

  1. giggle client for Linux
  2. Git Cola for Linux
  3. QGit Client for Linux

Running X-Window apps remotely

I’ve been using SSH (Secure shell) for a looooong time on my Linux based systems for remotely controlling my headless servers. During this week I had a guy over at my desk that wanted to have a look at some of my settings on my Ubuntu box to see the difference in configs between what he has done and what I have done differently. I quickly jumped into putty on my windows box (since that is my daily workstation because of VS2010 ) and connected to my server remotely.

Opened up nautilus on linux so I could visually browse to my config files (could have probably done this through shell  too), Nautilus then opened up in my windows OS as any other window would have. Since the other guy has never seen this he was obviously shocked to see the window appearing in my windows environment. I then went off to explain to him how this works and thought that I would share this since a couple of people have asked me “HOW DO YOU DO THAT” before, so here goes.

Remote-X (not the official name I think , but that’s what I call it ) is enabled via ssh or secure shell. SSH is a protocol used to securely access a Unix/Linux machines from another that allows you to run command line and graphical apps remotely and also for file transfers or creating secure networks over the internet to tunnel through(such as a remote http proxy or so). To accomplish this we would need 2 things.

  • Open SSH server for hosting and setting up the secure connection to the linux box that you want to connect to (Thus this runs on the server)
  • SSH client such as Putty to then connect to the server remotely via ssh protocol

Note that is a lot more uses for ssh such as secure ftp connections but I am just going to explain the basics to get a remote connection with openssh and remote X execution from a windows box.

Server Side Setup

First thing to-do is to get Open ssh setup on your Linux box (I am using Ubuntu 10.04 so my commands provided relates to Ubuntu).

Install Open ssh on your Ubuntu box via the following command

sudo apt-get install openssh-server

Once this is done you can change your configs as needed (have a look in the man file. man sshd_config for more info or just lookup sshd on the ubuntu site)

Now we just need to get the openssh daemon running and we’re set on the server side. This can be done with the following command.

sudo /etc/init.d/ssh restart
Client Side Setup

From a client side perspective , most machine would have a client to connect to a server via ssh such as putty. Ubuntu just needs the client downloaded to it and your set, since you have X-server doing all the X windows stuff on ubuntu by default, you won’t have a problem at all in opening the X-apps remotely that sits on your server.To get that client you can simply run the following command and then connect via that client once your set.

sudo apt-get install openssh-client

From a windows perspective, it is a tad more complicated because windows does not make use of the X- server natively for hosting or running its windowing system. In order to get that sorted we must do a couple of things as follows.

    • Download and install X-Ming a X-windowing server for windows.This will provide the capability to windows to run X-window apps within windows.
    • Download and install Putty used for connecting via ssh ( apparently X-ming has a way to connect to a server via ssh with its own mechanisms but I haven’t used that before. I prefer the familiar )

Once these 2 apps are installed you are all set to connect to your server.

Try Connecting ……

To check that the whole process as explained before was successful, we will try and connect to our Linux server over ssh.

    • Run xming and make sure you see the xming icon in your taskbar in the bottom right of your screen.

 

    • Now run putty and enter your server details

    • Before we connect we need to make sure Putty understands that we want to allow X-window apps to execute. Browse to the SSH settings in putty and expand that. You should see a sub option for X11, open that and then check that the X11 forwarding on the right hand is enabled.

    • You can now click “Open” and you should be connected to your remote server and prompted for the login details of that remote machine

.

    • Once logged in you can try and run a X-App to see that the remote server will then forward the X11 window to your windows machine.I usually run the following command to test: xcalc and will then be presentedwith the XCalc app in your windows environment

.

Sooooo , that’s quite a process on paper, but its actually very simple once you’ve tried it. You can then execute any X-window app remotely with convenience and this is very useful if you have multiple Linux of Unix machines that you work off.

Try this out if you have a remote scenario and let me know if you come right, if you struggle or have some questions.Drop me a comment or question and we’ll take it from there.

Happy computing. Ciao ciao

Scraping the world (How to screen scrape/rip images off the web)

Had to get some info on the web for one of my pet projects recently and found that there was no available web service on the web to provide the data to me that i was in search of. So i decided to look for sites that can provide me with up to date data and then scrape it off there for my own consumption (Not the nicest thing to do I know but it was the only way. Will not mention them at this point). The project consist of 2 parts namely

  1. the scraper service that mines the data and pushes only new relevant data into my data store
  2. The User interface to administer the data and consume it in a functional way.

The idea still hangs around consuming the data in a silverlight application or maybe utilizing asp.net with some other funky features, from WCF REST and DATA SERVICES but the focus is on rapid building as I want to make use of the app quickly but not spend to much time getting it up.

Tech used in scraper so far :

  1. Dotnet 4
  2. SubSonic (Probably one of the best open source projects i’ve used)
  3. Silverlight 4
  4. RegEx and Linq for parsing and getting the “STUFF” in the right format.
  5. SGML Reader Library by Mindtouch to get the HTML well formed
  6. Some other things that I maybe forgot to mention 🙂

Anyhoo, there has been some challenges in scraping the data as a lot of the sites did not comply with XHTML standards, thus making it difficult to consume the data (Hence SGML READER coming to the rescue there). String manipulation is ok to do but I wanted a structured way of working with the data I get back and be able to build queries with the use of linq. So I opted for getting the dat ain well formed XML and then working from there.

Eventually after pulling out the needed info from the xml into a workable object and then pushing that to my store, I moved on to the next task. The data contained URL’s to images that I needed to store within my DB as the APP should not rely on the online images , thus pulling the images off the web and serializing then to binary data and then storing them as an associated record item in the DB. DEFINITELY NOT AS EASY AS I THOUGH. But eventually got it right. Here is a little snippet of converting the image from URL to raw bytes and eventually back when retrieved from the data source.

(NOTE : This is a quick and dirty as I am purely building POC for myself so no guarantees that this will work for you. I am using Log4net so the “_log” references the global object of my application you can take that out and replace error handling in what ever way you need)

Image Manipulation Snippet
  1. ///<summary>
  2.     /// Downloads the image RAW.
  3.     ///</summary>
  4.     ///<param name=”Url”>The URL.</param>
  5.     ///<returns></returns>
  6.     public byte[] DownloadRawImage(string Url)
  7.     {
  8.       Image imageIn = this.DownloadImage(Url);
  9.       MemoryStream ms = new MemoryStream();
  10.       imageIn.Save(ms, System.Drawing.Imaging.ImageFormat.Gif);
  11.       return ms.ToArray();
  12.     }
  13.     ///<summary>
  14.     /// Converts the byte array to image.
  15.     ///</summary>
  16.     ///<param name=”byteArrayIn”>The byte array in.</param>
  17.     ///<returns></returns>
  18.     public Image byteArrayToImage(byte[] byteArrayIn)
  19.     {
  20.       MemoryStream ms = new MemoryStream(byteArrayIn);
  21.       Image returnImage = Image.FromStream(ms);
  22.       return returnImage;
  23.     }
  24.     ///<summary>
  25.     /// Downloads the image.
  26.     ///</summary>
  27.     ///<param name=”Url”>The URL.</param>
  28.     ///<returns></returns>
  29.     public Image DownloadImage(string Url)
  30.     {
  31.       System.IO.Stream webStream = null;
  32.       Image tmpImage = null;
  33.       int retries = 5;
  34.       while (retries > 0)
  35.       {
  36.         try
  37.         {
  38.           // Open a connection
  39.           System.Net.HttpWebRequest httpWebRequest = (System.Net.HttpWebRequest)System.Net.HttpWebRequest.Create(Url);
  40.           if (!string.IsNullOrEmpty(_proxyUser))
  41.           {
  42.             if (!string.IsNullOrEmpty(_proxyDomain))
  43.               httpWebRequest.Proxy.Credentials = new NetworkCredential(_proxyUser, _proxyPass, _proxyDomain);
  44.             else httpWebRequest.Proxy.Credentials = new NetworkCredential(_proxyUser, _proxyPass);
  45.           }
  46.           httpWebRequest.AllowWriteStreamBuffering = true;
  47.           // Request response:
  48.           System.Net.WebResponse webResponse = httpWebRequest.GetResponse();
  49.           // Open the data stream:
  50.           webStream = webResponse.GetResponseStream();
  51.           // convert webstream to image
  52.           if (webStream != null)
  53.             tmpImage = Image.FromStream(webStream);
  54.           // Cleanup
  55.           webResponse.Close();
  56.           retries = 0;
  57.         }
  58.         catch (Exception ex)
  59.         {
  60.           // Error
  61.           _log.Error(string.Format(“Exception caught in process: {0}”, ex.ToString()));
  62.           _log.Info(string.Format(“Retry in process for : {0}”, Url));
  63.           retries -= 1;
  64.         }
  65.       }
  66.      return tmpImage;
  67.    }

The amount of records I had to build from the data scraped is also quite a lot and takes a long time. My plan is to convert some of my scraping code to make use of the parallel framework in .net to see what the impact on the results will be.

Hope this is useful to someone.

Drop me a comment if you found this of interest or you maybe have something similar you are doing.

Super Duper Computing Power

Recently had the privileged to go into the room that houses one of the largest super computing clusters in South Africa, And I must say. After my visit in there I can think of all kinds of reasons why I would want my own :). Saw an IBM Rig and a CRAY rig, equally impressive but the cray just seems way better. It is absolutely magnificent pieces of hardware. Not only are they powerful but pretty too.

Here is a snap of how the CRAY machine looks like.(With complements of the cray site )

Fine piece of art that.

Did some looking around on the internet and it is quite astonishing what they do nowadays with supercomputers. These GIANT have been around for ages, but they primarily use these bad boys for large calculations or simulations or where processing power is key , and if you compare it to the power they had 20 years ago it is quite astonishing how far we have come.

Have a look at the cray site if you also find this interesting like me.
You can also look for more tech info on Wikipedia just out of interest.

L8ters
Ciao Ciao

Boot Menu edit or BCDEdit

I recently had to work quite allot with the MS BCDEdit tool to get my boot options sorted out for bootable VHD files and for changing options on Winserver 2008 to boot without a hypervisor running.( This was done so that I could get VMWare and Virtualbox running, since Hyper-V and the others cannot co exist) it was quite tedious doing everything in command line. It works but visualization of such function is always nicer , so I set out on looking for a tool that can assist me and found a nice one from the guys at NEO Smart

This is a visual BCDEdit tool , and it works absolutely marvellous  for changing those entries.

L8ter

Virtualization and Windows activation

Hey there again , and happy new year. This is the first post for 2010. Big year for us guys down here in South Africa.

I thought I’d share a little resolution I’ve struggled with this morning for people that make use of virtualization quite frequently.

I was installing windows server 2008 on VMware and run into an issue when activating the OS. My environment is setup using NAT for networking.
The issue is as follows:
Code:
0x8007232B

Description:
DNS name does not exist.
Not sure why this happens BUT the resolution was quite interesting.
You have to run an elevated command prompt, type “slmgr.vbs -ipk your-serial-activation-key”
This will take a while and you should be presented with a popup message when this is done.

Now you can try and reactivate windows. This worked wonders for me.

Hope this is help full. 🙂