In almost every technology we rely on today, there is some sort of protocol behind it. And if you’re an active techie, you’re likely to be greeted with this term more frequently. For example, if you wanted to transfer some files from one computer to another using a network connection, there is a protocol that can help you to achieve it.

So, what is a protocol?

A protocol is a set of standardized rules, procedures, and specifications that governs how data is transmitted over a network. What makes up the internet is data in form of videos, documents, text-based files, images, and many others. The way how we interact with certain data on the internet is governed by logical specifications and rules which have been standardized to ensure smooth packet flow.

To further understand how network protocols work, we need to go back in time to where everything started.

In the early days of computing and packet-switched networking, there was a need for a better way to facilitate the exchange of data between different devices and software applications.

The industry was getting occupied with various players like software developers and device manufacturers who were making various types of products. And so, the need to standardize how emerging platforms interacted and communicated with each other was so critical.

This meant that any software or device would be designed to work based on standardized protocols or follow a set of logical rules and specifications while communicating with each other, even if they were developed by different manufacturers or developed in different programming languages.

Why are protocols important?

There are hundreds of network protocols, but most of them have been standardized to serve specific applications and use cases. The truth is, protocols are crucial in the following ways:

Standardizing interoperability

Without protocols, every device or application would have been designed to communicate directly with every other, which would be impractical and unsustainable as the number of devices and applications is ever-increasing. This would have led to the lack of interoperability, making it difficult to share data and resources and limiting the potential for innovation and progress.

Defining how data is packaged and transmitted

Protocols define how particular data is packaged by specifying the format and structure when being transmitted. This includes information about the type of data being transmitted, the size and structure of each data packet, the order in which packets are transmitted, and how errors are detected and corrected.

Identifying and Addressing data

When data is transmitted between devices, it is packaged into packets, each of which includes both the source and destination addresses. Typically, protocols use IP addresses, or MAC addresses, to identify the source and destination of data packets. These unique identifiers are key to distinguishing data packets sent from one device to another across a network, even if there are multiple devices connected to the same network.

Standardizing data security

The lack of standardization in the way data is transmitted would have made it difficult to ensure the reliability and security of data packets sent over a network. This would have resulted in a high incidence of data loss, corruption, and security breaches, undermining the functionality and usefulness of existing technologies.

Fortunately, protocols provide a broad framework, critical to data security in the applications through the following ways;

• Defining standards for Authentication and Authorization: Protocols such as TLS/SSL and SSH provide secure communication by authenticating the identity of the communicating parties and ensuring that only authorized parties can access the data.

• Implementing Encryption: Protocols like HTTPS, SFTP, and FTPS provide data encryption during transmission to ensure that sensitive data cannot be intercepted by unauthorized parties.

• Mainstreaming Data Integrity: Protocols like MD5 and SHA provide integrity checks to ensure that data has not been tampered with during transmission.

• Access Control: Protocols such as LDAP and RADIUS provide centralized access control mechanisms that allow administrators to control user access to resources.

• Auditing and Logging: Protocols like Syslog provide auditing and logging facilities that allow administrators to track system activity and detect potential security breaches.

Who defines Network protocols?

Network protocols are typically defined by standardization bodies or organizations, such as the Internet Engineering Task Force (IETF) or the International Organization for Standardization (ISO). These organizations develop and publish specifications and logical procedures that describe how protocols are used to facilitate communication between different devices and systems over a network. In some cases, network protocols may also be developed by individual companies or groups, but these are often based on existing standards and are subject to review and approval by relevant standardization bodies.

How are Protocols implemented?

At the hardware level, devices need to have the necessary physical components, such as network interface cards, modems, or wireless transceivers, to send and receive data according to the protocol. These components are designed to follow the specifications of the protocol and perform tasks such as packet switching, error checking, and flow control.

At the software level, developers write code that adheres to the rules and standards set forth by the protocol. This code ensures that the device or software can communicate with other devices and software that also follow the same protocol.

Now, almost all internet-based applications rely on protocols to facilitate the smooth transmission of data from the source to the target destination. In this article, let’s try to understand how some protocols work in various applications we use;

Fetching webpages using HTTP

Every webpage you access on the internet is hosted and stored somewhere on a physical computer, probably located far away from you. We use browsers to fetch and access web pages. For example, when you visit ugacomp.com, you’re telling the computer hosting this website to give you access to its web pages. For the end user, this entire process may seem so easy. But behind the scenes, there is the underlying logic that defines how these webpages are fetched from the server or remote computer hosting them. This logic is activated by a defined and integrated protocol in the browser called Hypertext Transfer Protocol (HTTP).

The HTTP protocol is primarily designed as the standard logic for requesting and transferring webpages from a web server to the client using browsing software like Google Chrome, or Microsoft Edge. So, this works in such a way that when you type a URL into your web browser, it means that you’re sending what we call an HTTP request to the web server, asking for the web page you want to see. The web server then reacts back with what we call HTTP response by providing the requested web page so it can be displayed in the browser.

RECOMMENDED READING: How to deploy WordPress on a Ubuntu LAMP Server

HTTP Protocol Specifications

Protocols are defined as specifications, outlining the rules, procedures, and formats used to establish communication between different devices or systems. These specifications typically describe how data is exchanged between devices, what types of data are supported, and how errors and conflicts are handled.

Protocol specifications are important because they ensure that devices or applications are designed to be interoperable in some aspects. Without standard protocols, it would be very difficult for different devices and applications to exchange data or work together in a coordinated way.

For example, the following are some of the HTTP Protocol specifications you should know about;

• Request and Response Messages: HTTP uses request and response messages to communicate between clients and servers. The request message typically contains a request method, URI, and headers, while the response message typically contains a status code, headers, and a message body.

• Uniform Resource Identifier (URI): A URI is used to identify a resource on the web. A URI typically consists of a scheme, host, port, and path.

• Request Methods: HTTP defines several request methods, including GET, POST, PUT, DELETE, and HEAD. Each method is used for a specific purpose.

• Status Codes: HTTP defines several status codes to indicate the status of a request. Status codes include 200 OK, 404 Not Found, and 500 Internal Server Error.

• Connection Management: HTTP uses a connection-oriented protocol, meaning that a connection is established between the client and server for each request. HTTP/1.1 allows for persistent connections, where multiple requests can be sent over the same connection.

• Headers: HTTP uses headers to pass additional information between clients and servers. Headers can be used for authentication, caching, compression, and other purposes.

• Cookies: HTTP uses cookies to store information on the client side. Cookies can be used to maintain user sessions, personalize content, and track user behavior.

• Security: HTTP can be secured using Transport Layer Security (TLS), which encrypts data between the client and server. HTTPS (HTTP Secure) is the secure version of HTTP and is widely used on the web.

How to implement HTTP Protocol

So, to implement the HTTP protocol, developers write code that follows its defined specifications, rules, and standards. The browser’s code sends an “HTTP request” with specific headers and data, and the server’s code responds with an “HTTP response” with its own headers and data. Both the browser and server must follow the same protocol for communication to be successful.

For example, here is a simple code snippet in Python that sends an HTTP GET request to a server and receives a response:

import socket

# create a socket object
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

# get the server's IP address and port
server_address = ('www.example.com', 80)

# connect to the server
s.connect(server_address)

# send an HTTP GET request
request = b'GET / HTTP/1.1\r\nHost: www.example.com\r\n\r\n'
s.sendall(request)

# receive the response
response = b''
while True:
    data = s.recv(1024)
    if not data:
        break
    response += data

# print the response
print(response.decode())

This code imports the socket module and then performs several actions to send an HTTP GET request to a server, receives the response, and prints it. Below is a line-by-line breakdown of this code:

The following line imports the socket module, which provides low-level networking primitives for creating and interacting with network sockets.

import socket

The line below creates a new socket object, s using the socket() method. The arguments socket.AF_INET specify that the socket should use the IPv4 address family, and socket.SOCK_STREAM specifies that the socket should use the TCP protocol.

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

This line defines a tuple server_address, which contains the address of the server to connect to. In this case, the address is the hostname www.example.com , and the port number 80.

server_address = ('www.example.com', 80)

This line connects the socket s to the server at the address server_address.

s.connect(server_address)

These two lines send an HTTP GET request to the server by encoding the request string as a bytes object and using the sendall() method to send the bytes to the server.

request = b'GET / HTTP/1.1\r\nHost: www.example.com\r\n\r\n'
s.sendall(request)

These lines receive the response from the server by repeatedly calling the recv() method on the socket s and appending the received data to the response variable until all the data has been received (as indicated by the if not data condition).

response = b''
while True:
    data = s.recv(1024)
    if not data:
        break
    response += data

Finally, this line prints the response received from the server after decoding it from bytes to a string using the decode() method.

print(response.decode())

Remote File Transfer using FTP

Transferring files from local computers to remote computers is possible because of the specifically established protocol called File Transfer Protocol (FTP)

This process involves an FTP client and an FTP server. An FTP client is a program that runs on your computer and allows you to connect to an FTP server. It provides a user interface to help you navigate the server’s file directory, select files to transfer, and initiate file transfers.

An FTP server, on the other hand, is a program that runs on a remote computer and listens for incoming FTP connections. When an FTP client connects to an FTP server, it provides authentication information (such as a username and password) to access the server’s file directory.

Once the FTP client is authenticated, it can navigate the directory tree of the FTP server and select files to transfer. It can then initiate a transfer request to the server, which starts transferring the selected files over the network.

RECOMMENDED READING: Step-by-Step Guide to Installing FTP on Ubuntu Server

FTP Protocol Specifications

The following are some of the key specifications that have to be considered when implementing the FTP protocol;

• Port numbers: FTP uses two port numbers, one for the control connection and the other for the data connection. The default port numbers are 21 for the control connection and 20 for the data connection

• Connection establishment: FTP uses the TCP/IP protocol for connection establishment. A client initiates a connection to the server on the control port, and once the connection is established, the client sends commands to the server to request data transfer.

• Authentication: FTP uses various authentication mechanisms to verify the identity of the user. The most common mechanism is the username and password. The server verifies the user’s credentials before allowing access to the files.

• Commands: FTP uses a set of commands for controlling the transfer of files. The commands include retrieving a file from the server, uploading a file to the server, creating a directory, deleting a file or directory, and so on.

• Transfer modes: FTP supports three transfer modes: ASCII, binary, and EBCDIC. The ASCII mode is used for transferring text files, the binary mode is used for transferring non-text files, and the EBCDIC mode is used for transferring files between systems that use the EBCDIC character set.

• Data transfer: FTP uses the data connection for transferring files. The server opens a data connection to the client for sending the requested file, and the client opens a data connection to the server for uploading a file. The data transfer can be in either active mode or passive mode.

• Error handling: FTP includes a set of error codes and messages that the server uses to notify the client of any errors during the transfer process. The client can use these error codes to determine the cause of the error and take corrective action.,

• Security: FTP is vulnerable to various security threats, including eavesdropping, data tampering, and data theft. To mitigate these threats, FTP can be used with Secure Sockets Layer (SSL) or Transport Layer Security (TLS) encryption.

How to implement FTP protocol in an application

Here is a sample code that demonstrates how the FTP protocol is implemented to facilitate file transfer in a Python application:

import ftplib

# establish FTP connection
ftp = ftplib.FTP("ftp.example.com")
ftp.login("username", "password")

# navigate to remote directory
ftp.cwd("/remote/directory/")

# upload a file to the remote server
with open("localfile.txt", "rb") as f:
    ftp.storbinary("STOR remote_file.txt", f)

# download a file from the remote server
with open("localfile.txt", "wb") as f:
    ftp.retrbinary("RETR remote_file.txt", f.write)

# close FTP connection
ftp.quit()

Let’s break down the above code line by line;

The following line imports the ftplib library that provides FTP client functionality.

import ftplib

This line creates an FTP object named ftp that connects to the remote FTP server located at ftp.example.com.

ftp = ftplib.FTP("ftp.example.com")

This line logs in to the remote FTP server using the provided username and password credentials.

ftp.login("username", "password")

This line changes the current working directory to /remote/directory/ on the remote FTP server.

ftp.cwd("/remote/directory/")

This line opens a local file named localfile.txt in binary mode for reading and transfers it to the remote server with the filename remote_file.txt using the storbinary method of the ftp object.

with open("localfile.txt", "rb") as f:
    ftp.storbinary("STOR remote_file.txt", f)

This line opens a local file named localfile.txt in binary mode for writing and downloading the file named remote_file.txt from the remote server using the retrbinary method of the ftp object.

This line closes the FTP connection.

ftp.quit()

Live-streaming using RTMP

RTMP stands for Remote Transport Messaging Protocol, which is one of the commonly used protocols for live-Streaming. In fact, Platforms like YouTube, Twitter, Instagram, Twitch, Facebook, and more use this protocol, allowing their users to publish RTMP streams.

RECOMMENDED READING: How to set up Nginx RTMP Server on Ubuntu to Live stream videos

RTMP Protocol Specifications

The following specifications are important to know when implementing RTMP protocol:

• Server-side implementation: You need to implement the RTMP server on the server side to receive and process incoming client requests. The server should be capable of handling multiple client connections simultaneously and managing streaming sessions.

• Client-side implementation: You need to implement the RTMP client on the client side to initiate the streaming session with the server. The client should be able to establish a connection with the server, send control messages and data, and receive media streams.

• RTMP message format: RTMP uses a binary message format to transmit data and control messages. Each message consists of a header and a payload. The header contains information about the message type, length, and timestamp. The payload contains the actual data or control message.

• RTMP message types: RTMP supports several types of messages, including protocol control messages, media messages, and shared object messages. Protocol control messages are used for handshake and session management, media messages carry audio, video, or metadata, and shared object messages allow clients to synchronize data.

• Handshake: RTMP uses a three-way handshake to establish a connection between the client and server. The handshake involves sending and receiving several control messages to negotiate the connection parameters.

• Streaming: RTMP supports two types of streaming: live streaming and on-demand streaming. In live streaming, data is transmitted in real-time, while in on-demand streaming, data is stored on the server and transmitted to the client on request.

• Security: RTMP supports several security features, including SSL/TLS encryption, user authentication, and access control.

• RTMP variants: There are several variants of RTMP, including RTMPT, RTMPS, RTMPE, and RTMPTE. RTMPT uses HTTP as a transport protocol, RTMPS uses SSL/TLS encryption, RTMPE uses encryption, and RTMPTE uses tunneling.

How to implement RTMP protocol in an application

The following is a sample application that follows some of the specifications required to implement RTMP

const rtmp = require('rtmp-js');

// create an RTMP connection
const conn = rtmp.createConnection({
  host: 'server_address',
  app: 'application_name',
});

// set up handshake
conn.on('connect', () => {
  console.log('Connected to RTMP server');
  conn.doHandshake();
});

// handle handshake response
conn.on('handshake', () => {
  console.log('Handshake completed');
  // create a stream
  const stream = conn.createStream();

  // send data over the stream
  stream.write('Hello, RTMP!');

  // close the stream and connection
  stream.close();
  conn.close();
});

// handle incoming messages
conn.on('message', (message) => {
  console.log('Received message:', message);
});

// handle errors
conn.on('error', (error) => {
  console.log('Error:', error);
});

// connect to the RTMP server
conn.connect();

Here is the quick breakdown of the above code;

This line declares a constant rtmp and assigns to it the value of the require() function which loads the ‘rtmp-js‘ module.

const rtmp = require('rtmp-js');

This line creates an RTMP connection using the rtmp module’s createConnection() method. It takes an object as an argument, which specifies the host address and the name of the application to connect to.

const conn = rtmp.createConnection({
  host: 'server_address',
  app: 'application_name',
});

This code sets up an event listener that waits for the connect event to occur. When the connection is established, the callback function logs a message to the console and calls conn.doHandshake() to initiate the handshake process.

conn.on('connect', () => {
  console.log('Connected to RTMP server');
  conn.doHandshake();
});

This code sets up an event listener that waits for the handshake event to occur. When the handshake is completed, the callback function logs a message to the console and creates a new stream using conn.createStream(), writes data to the stream using stream.write(), and then closes the stream and connection using stream.close() and conn.close() respectively.

conn.on('handshake', () => {
  console.log('Handshake completed');
  const stream = conn.createStream();
  stream.write('Hello, RTMP!');
  stream.close();
  conn.close();
});

This code sets up an event listener that waits for the message event to occur. When a message is received, the callback function logs a message to the console along with the contents of the message.

conn.on('message', (message) => {
  console.log('Received message:', message);
});

This code sets up an event listener that waits for the error event to occur. When an error occurs, the callback function logs an error message to the console along with the error object.

conn.on('error', (error) => {
  console.log('Error:', error);
});

This line initiates the connection process by calling the conn.connect() method.

conn.connect();

Conclusion

The primary goal of this article is to give you a picture of how protocols work and are implemented in applications. We talked about protocols like HTTP, FTP, & RTMP. We used these for example purposes, and there is a long list of protocols that we can’t exhaust. You can further make more research to help you understand everything.