How to access applications & websites behind Proxy Server from CLI or browser

In today's world, it's very common, industry standard practice to use proxy servers to safeguard their corporate and institutional security, to control employee internet usages, to balance traffic on the server or for faster speeds and bandwidth saving. Therefore to access the applications, API endpoints, blocked websites like GitLab, GitHub, Facebook, Youtube etc, that are behind a proxy server, one need to use the appropriate proxy server.

Today we would discuss, how to access applications/websites behind proxy server using CLI & browser in Windows and Linux systems? After going through this article, one would have the answers of below questions:

• Unable to pull the Git repository hosted on GitLab or GitHub using CLI i.e. from command prompt or Git Bash
• Unable to download packages from internet through CLI using wget, curl, apt-get, yum or pip
• Unable to access API endpoints, project applications in the development, qa, staging environment from the browser or CLI
• Unable to access blocked websites (GitLab, GitHub, Facebook, Youtube etc) using browser

Table Of Contents

1. Scope of the article
2. Prerequisites
3. Use Proxy Server while executing commands in the CLI
4. Use Proxy Server in the browser
5. Conclusion

1. Scope of the article

• Primary focus is to show, how to use the proxy server in Windows and Linux systems through http_proxy environment variable to make applications accessible from CLI 
• Would also talk about using the proxy server in the browser either by adjusting the proxy option setting or by using different browser add-ons for proxy switching

Note: For simplicity of this article, suppose we have a HTTP proxy server on the network between our host machine and the GitLab server and we want to access the GitLab console from browser and checkout the git repository through CLI.

2. Prerequisites

One have to find out which applications/websites are behind which proxy server and their corresponding details that means:

• Proxy Server IP address along with the protocol (i.e. URL) 
• Port Number 
• For password protected proxy server, username & password

3. Use Proxy Server while executing commands in the CLI

One need to to set the http_proxy environment variable in order to checkout the git repository through CLI or to download packages from internet using wget, curl, apt-get, yum or pip.

Few important things, one should keep in mind while setting up http_proxy :

• Steps of setting up http_proxy depends upon the type of the OS
• Decide wisely, whether wants to set proxy variable in persistent or in non persistent way

Let's start our journey.

Format of http_proxy

https_proxy is set with hostname or IP address of the proxy server in URL format:

## Using hostname 
http_proxy=http://hostname:port
Example: http_proxy=http://proxy.anshulgnit.blogspot.com:8080

## Using IP addres
http_proxy=http://ip_address:port
Example: http_proxy=http://10.135.15.225:8080

## For password protected proxy server 
http_proxy=http://username:password@hostname:port
Example: http_proxy=http://username:password@proxy.anshulgnit.blogspot.com:8080

Note: 
- If password contains special character like @ the encode it accordingly. Eg. @ -> %40
- If protocol used for proxy server is https then accordingly set https_proxy variable

Setting http_proxy in Linux / Unix / Mac OS

We would discuss everything with respect to bash shell and their respective profiles. One can set http_proxy persistently or non persistently or for a specific user or for all users. 

1. Set http_proxy in non persistent way
   

   $ export http_proxy=http://proxy.anshulgnit.blogspot.com:8080
   $ export http_proxy=http://10.135.15.225:8080
   $ export http_proxy=http://username:password@proxy.anshulgnit.blogspot.com:8080
   
   ## through command line option in wget, curl
   $ wget --proxy-user=username --proxy-password=password http://hostname/some-file-resource
   $ curl --proxy-user username:password http://hostname
   

2. Set http_proxy in persistent way
   - set http_proxy specific to a user
   One need to add the command to the appropriate profile file for a shell. For example: for bash shell, it is .bash_profile or .bashrc file, located in the home directory
   

   ## Open the .bash_profile 
   $ vi $HOME/.bash_profile or vi /home/anshul/.bash_profile
   
   ## Add the below lines, change it accordingly if not password protected 
   https_proxy=http://username:password@hostname:port
   export $https_proxy
   

   - set http_proxy for all users
   

   ## Open the /etc/profile
   $ vi /etc/profile
   
   ## Add the below lines, change it accordingly if not password protected
   https_proxy=http://username:password@hostname:port
   export $https_proxy

Setting http_proxy in Windows

The syntax is pretty much the same as it was for Linux. The only difference is in the command therefore we haven't repeated the same thing again. In windows, one can also set http_proxy variable using GUI just like one add JAVA_HOME variable.

1. Set http_proxy in non persistent way
   

   ## set 
   C:\Users\Attitude>set http_proxy "http://hostname:port"
   ## verify 
   C:\Users\Attitude>echo %http_proxy%
   

2. Set http_proxy in persistent way
   

   ## set 
   C:\Users\Attitude>setx http_proxy "http://10.135.15.225:4128"
   SUCESS: Specified value was saved.
   ## verify 
   C:\Users\Attitude>echo %http_proxy%
   http://10.135.15.225:4128
   
   ## wants to clone repository using Git Bash
   ## set
   $ git config --global http.proxy http://10.135.15.225:4128
   ## verify 
   $ git config --get http.proxy

4. Use Proxy Server in the browser

Each browser has an option for setting up proxy server to make applications & websites accessible from web browser that are behind these proxy servers. Here also, we have two use cases:

• Everything i.e. all the required applications & websites are behind a single proxy server
  In this case, it's a one time activity to set up the proxy server by changing the proxy option setting in the required browser.
  
  For Chrome (Version 57.0.2987.110):
  Settings -> Type "proxy" in search box -> click on "Change proxy settings.." button -> select "Connections" tab in the "Internet Properties" window -> click on "LAN settings" button -> check "Use a proxy server for your LAN" checkbox -> provide proxy server IP address & port then click "ok" button to save the changes.
  
  
  
  
  Note: By default, Google Chrome will use the proxy settings that Internet Explorer uses. For IE (Tools[Alt+X] -> select "Internet Options", same "Internet Propeties"window would open but check the name of the window, it would be "Internet Options".)


• Multiple environments (dev,qa,staging etc) & multiple applications (API endpoints, GitLab, GitHub etc) are behind different proxy servers
  In this case, it's not feasible to change proxy server setting from browser, every time one switches from dev to qa or from qa to dev or wants to access the GitLab or GitHub. Therefore it's advisable to use browser add-on for proxy switching like Proxy Switcher, Proxy SwitchySharp etc.
  
  
  

5. Conclusion

It's very crucial to know how to bypass the proxy server in order to access application behind the proxy server. http_proxy environment variable should be set accordingly to access resources behind the proxy.

Thank you for reading this article! Hoping that now you can easily resolve the proxy server error. I am happy to hear from my readers. Any thoughts, feedback, critique - all welcome! Happy Learning !!