By Taylor Centers | June 7, 2021
Danger Ahead!
Congratulations – you have deployed a new product, device, or server that runs on your customer’s premise. The product development lifecycle, however, does not end at deployment. Support and maintenance are key components of delivering a robust product. When a customer encounters an issue with a deployed device, there are a few options: 1) attempt to coach them through troubleshooting over the phone (or any telecom system), 2) send an expert to be on-site with the customer and the problem device, or 3) create a remote access system in the product that allows your experts to access the device from anywhere at any time.
From a product and business perspective, being able to access a remote system from anywhere at any time is the obvious choice, care must be taken – any remote access solution has the potential to be utilized by hackers as well. Having a secure connection to your deployed device is possible, and we’re here to help – so in this blog post, we review some important considerations in designing remote access systems for deployed devices.
Purpose and Background
Remote administration is an attractive feature to add to any product that lives at your customer’s location. Support is a critical part of the product lifecycle and a remote administration ability allows your support team to be responsive, agile, and efficient. Unfortunately, there is a risk versus reward calculation that must be made when considering adding a remote administration capability to your product. It is a security minefield where a mistake in the tool implementation, network configuration, credential management, or software patching can lead to not only the compromise of your product, but the customer’s infrastructure. Understandably, this kind of software is an attractive target for attackers, and there is a history of exploitable bugs in many of the protocols used for remote access including Virtual Network Computer (VNC) and the Remote Desktop Protocol (RDP).
The presence of remote access software as well as the network configurations required for it mean an attacker may have a way into the protected network. Security professionals often refer to Remote Administration Tools (RATs) as rootkits, implants, or Command and Control (C&C) software. This is a bit tongue in cheek because it paints these official tools as malicious software, but there is some truth in the designation. A RAT and a C&C tool can behave similarly. They perform the same functions - giving a remote user privileged access to the local machine. There are a lot of pieces to setting up a secure remote administration ecosystem, any one of which can potentially be a target for attackers.
Authentication and Authorization
Perhaps the most important consideration is deciding who has access to the target system, and how they get access. This is controlled through authentication – making sure the correct person has access to the system. There are different ways that a user can authenticate to the target system, which could include a username and password, or with a pre-shared key, where the user has one half, and the other is set up on the target. This problem, as with most, becomes increasingly complex with added scale. Managing the pre-shared keys for one device is a different challenge than managing keys across hundreds or even thousands of devices. Additionally, applying security best practices at that scale increases the burden. A risk of re-used passwords or keys means a weak password or leaked key will result in every device being compromised. If an employee leaves the company or switches teams, or a new employee joins, there needs to be a mechanism for adding and removing their accesses. Some industries have compliance standards that must be met - for example, requiring any access to be logged and associated with a specific individual. This means that a single shared “technician” or “support” account is not sufficient. Also, certain types of systems that handle special information require Two Factor Authentication (2FA) to better secure access and protect the system.
Infrastructure
The amount of infrastructure required to support authentication for a remote administration system can vary, and typically scales with requirements for revocation, rotation, scalability, and risk mitigation. A device can be pre-configured with a default username and password without the need for any Lightweight Directory Access Protocol (LDAP) or Single Sign-on (SSO) infrastructure. However, this affects the ability to scale the number of users, revoke access, and increases the risk of a leaked password. The same can be said for pre-sharing an SSH key, although this has the benefit of being significantly harder to guess or brute force.
More advanced infrastructure leverages Pluggable Authentication Modules (PAM) which communicate to SSO and LDAP infrastructure to authenticate users. A system using LDAP and PAM can also share users RSA or Elliptic Curve keys with the remote system. User access and management can then be done from the SSO system and allows users to update credentials and keys.
Deployment
To support using an SSO, remote devices will need to be configured with a PAM to communicate with SSO. This is additional software to be built into and deployed with the remote device. Without this software, all remote users’ credentials will need to be preconfigured on the devices prior to deployment and any updates to the credentials will need to be made by Software Update which can be a challenging process itself.
Revocation & Rotation
A hardcoded username/password or key cannot be easily revoked or rotated, these are credentials shared with the system at build or deployment time and are only managed on the device itself. To remove a user’s access to the devices, each remote system will need to be reconfigured by an administrator. As the number of remote devices grow this solution becomes infeasible. Custom software can we be written to automate the process, but is predicated on having singular access to all devices which itself perpetuates the problem. Centrally managing credentials with LDAP or SSO allows for grating access to certain groups of users, setting rules for rotating credentials, and enables administrators to revoke access to compromised accounts.
Scalability
When considering scalability, we think of the growth of both the number of users who will access the remote system, and the number of remote systems there are to access. As either of these numbers grow the risk grows as well. A solution that enables the team to manage authentication centrally is critical.
Risk
The primary risk of remote administration systems is for malicious users to gain access to the system. Though sophisticated attackers will spend months finding zero-day vulnerabilities in the software, making sure your devices are up-to-date on the security patches is a reasonable preventative measure. However, most attackers will gain access to the system through leaked or stolen credentials. The two largest concerns are being responsive to compromised credentials and limiting the affected systems by a compromise.
Network Configuration
There are many options for configuring the network so that a remote user can reach the target system, and this will depend heavily on the existing network infrastructure for the deployment. Typically, a device or server deployed at a customer location is bound by the network policy of the customer. It is a challenge for both the vendor and customer when requirements for the network and firewall rules need to be changed to support the new system. There are ways to open access to target devices without any burden on changing policy. This involves creating a tunnel out from the local device to the remote server. Generally this is done by either establishing a reverse SSH tunnel or having the device join a Virtual Private Network (VPN). Both of these methods reduce the risk of a remote administration system by not having the target open to any incoming connections. However, managing either of these tunnels at scale is a challenge and both require configurations which are robust enough to reconnect or fallback when something goes wrong. Reliability is particularly pertinent when the customer is already having difficulties with the device and needs remote access to function properly for support.
Infrastructure and Vendors
Additional internal infrastructure will need to be stood up and supported to enable managing and using remote administration tools. This includes jump-boxes, a VPN, Public Key Infrastructure (PKI), front end software, logging software, monitoring and diagnostic tools. Some vendors offer packaged remote administration software. You can find companies whose offerings range from the complete solutions to focusing on specific parts. While paying for this solution when there are many open source tools available that it is possible to set up and manage a system in-house may seem expensive, often the cost is merely just more transparent. These companies specialize in building this software and making it accessible and scalable. These companies can respond to vulnerability advisories and disclosures and provide consistent upkeep and support. In comparison, relying on free open source tools may require more vigilance from a system administration team.
Conclusion
Relying on a third party does mean giving away a measure of trust, requiring their own balance of considerations if they can meet your needs. There are vendors based around the world, so information may pass through foreign servers. The privacy policy and use of data varies between companies and may be a cause of concern for certain types of information. Some companies only support certain Operating Systems or have other technical requirements (e.g. requiring user interaction to establish the connection) which may not be suitable for your use case. And finally, the reputation of the company will reflect on your product. At River Loop Security, we have seen, used, and developed many different remote administration tools. These are typically one of the highest areas of concern when we work with our clients when either coaching them through the design of their own system (both architectural design and technical implementation) or when performing a device risk and vulnerability assessment. There are a lot of pieces that need to work together to meet the technical and compliance criteria and we can help. The benefits of having a remote administration capability are evident and overwhelming - you should have one. We just want you to be safe while doing it. Contact us any time if you need help designing or testing the security of remote administration systems for connected devices.
PS: Stay tuned for a subsequent blog posts that will take a deeper look into the technical implementations of a remote administration ecosystem with code, proof of concepts, and a look at how Public Key Infrastructure (PKI) can be used.