Tag: private server

Real time body camera system – Server Storage – part 3

January 3, 2022 /

In the last two parts of this series, we discussed our network protocol and the architecture of our body camera system. We shall discuss our backend recording and streamers service in this final part. After that, we shall present the budget we burnt for this MVP, and finally, we shall discuss why it did not work.

There are multiple server video streaming solutions across the market. Unfortunately, most of them require installing new desktop software or plugins. At the same time, we saw that no video storage format is codec agnostic and could support multiple frames using different codecs. All these weaknesses forced us to develop our storage format for our videos. After a reasonable amount of time of thinking about what would be the format we need for this kind of work, we formulated the following requirements:

  • Blazing fast append: We wanted to write down the incoming frames as fast as possible. Every slowdown of reindexing the frames would decrease performance and increase the cost.
  • File-based:  Storing a significant amount of data into a database is the wrong approach for media-based files. So the file format had to be binary based. Because of the previous requirement, we had to skip the index. Every index recalculation would end up in a lousy performance.
  • To support telemetry: We did not stream only video and audio data, but we also streamed telemetry. There had to be a way to stream its frames as well. Plus, there were use cases in which the user of the body camera could want to stream only telemetry, but not video.
  • Websocket streaming: Since we can not support another streaming format, we decided that streaming from our server to the web browser clients in the headquarters will be based on WebSockets. To do this properly, we had to implement our video player.

Fortunately, if we analyze our network protocol, we can see the following characteristics which will fulfill the requirements:

  • Partitioning: Every network packet has a unique user id and date. And that is enough to determine the filename of the stream uniquely.
  • Counter: Every network packet has a unique counter value, and this value is attached to the date. At the beginning of every day, we moved the counter to zero. If we analyze further the logic of this counter, it could be used as an index key, by which we can sort by time the whole stream.
  • Supports telemetry: Our network protocol supports telemetry by default.
  • Supports WebSockets: We can reuse the same binary message we received from the Android device for WebSocket streaming. The message must be just encoded properly for WebSocket streaming.
You can see the modified version of our frames container on the diagram. It follows the header:body format, where the header represents a metadata structure for the next frame. By iterating the whole file, we could index the metadata into our server’s memory

With the information from the previous bullets, we can define the following logic. We append every incoming packet to its corresponding file on the filesystem, similarly to what pcap is doing with the network packets. At the same time, another process is reading the file and building the index in memory of our service. And the service uses this index to restream the recorded network packets through web sockets to the web browser player.

To implement the described logic, we decided to build the following system modules:

  • UDP server listener module: The idea of this module is to listen for UDP packets and reroute them to a concurrent queue. The FileWriter module later consumes this concurrent queue.
  • Concurrent queue module: Having in mind that we can have multiple process workers, instead of using mutexes or any other synchronization mechanisms, we decided to communicate using queues between the processes.
  • FileReader module: This module’s primary duty is to read the file packet by packet, using the already loaded index.
  • FileWriter module: The idea of this module is to take the packets from the queue and store them into the file. Partitioning per file per queue was implemented, and every file had a FileWriter process.
  • Indexer module:  It reads the file and indexes the network packets into the memory. After that, it is used by the Streamer module to stream data.
  • Streamer module: This was a collection of processes that started by given offset and used the indexer module to send data to the WebSocket server.
  • Web browser player module: The module was used to decode the network packets coming from the WebSocket server and play video and telemetry data in the browser.
  • Synchronization module: The idea of this module was to provide a way for the synchronization of missing packets between the Android device and the streaming server. We used the index module to return a given user and date for which frames are missing.

One can easily modify the proposed architecture to support cloud deployment and high scalability by replacing the concurrent queues with message brokers and the local filesystem with GlusterFS.

You can see a sample system architecture on the diagram describing the listed components. Arrows represent the data flows and how the data passed through the system

After we finished the technical details of the implementation, let’s discuss how much it cost for us to implement the MVP:

Budget:

  • Android device (200$): We decided to use a standard Redmi 3 device. It supported all the needed features, and it had an excellent battery.
  • Extended battery development (3000$): We developed two battery versions because the first one was not good enough, and our hardware vendor did not provide a quality job. We had to switch vendors, etc.
  • USB Camera (200$): Fortunately, we used an already produced board, so the price was relatively low. Still, we had to buy multiple cameras until we found the most suitable one.
  • 3d printing (400$): 3d printing of multiple prototypes is expensive. And we had to try with various variations.
  • Camera mounts (30$): The camera mounts were already manufactured.
  • Software development (23000$): One developer spent a whole year working part-time on this project. He implemented the backend server and the mobile application for the Android device.
  • Hardware development (8000$): Our hardware guy spent a couple of months developing proper housing and an alternative battery unit for our Android device.
  • Business development (1500$): Fortunately, we did not spend a lot of money on business development.

So we managed to implement the technical MVP for a total cost of 36330$. We tried to sell it, and we failed brutally.

Why we failed

As a team without experience in developing hardware products, we made many errors. Fortunately, it was our first and last try at selling a hardware product. We took our lessons, which I shall list:

  • No business need: For every successful product, you need a local business need, with which you can test your idea and see whether you will have traction. Burgas is an almost crime-free city, so no need for such a system.
  • No hardware ecosystem: There is no ecosystem of electronic hardware manufacturers in Burgas. So you become dependent on people you do not know and even have never met.
  • No delivery pipelines: Making hardware depends on your components delivery pipelines. We did not have any established channels and no partners with such.
  • No investor: Hardware startups are not for bootstrapping. You need a good amount of money to hire the proper people and to make sure once you have MVP, you can buy a good amount of items. Hardware items supply can end, and after that, you have to redesign your solution.
  • Wrong paradigm: Hardware products do not scale so much as digital ones. It will help if you have a good global distribution network and marketing to do it successfully. Being in the 4th city by size in Bulgaria, with 200,000 people, did not help.

In conclusion, despite the problems, we managed to produce MVP, which is a piece of fantastic news. Unfortunately, we could never sell this MVP to anyone for the listed reasons. Looking at the good side of things, we learned what mistakes to avoid when penetrating a market. It helped us with our following products. 

Cybersecurity tactics for small teams – Business Security

November 6, 2021 /

In the last couple of months, we discussed how you could achieve a good level of personal security for your team members. We covered the topics of physical information security, home network security, and finally, your hardware devices cybersecurity. With this article, we shall cover the issue of how you can upgrade your cybersecurity defenses as a team. The article will cover the necessities of remote-first groups because they are harder to defend. You can use the same approach to protect your office or shared space-based team. Still, the focus will be on underfunded small groups. At the end of the article, I shall present a sample budget for your team infrastructure.

But before going to the budget, let’s analyze how a remote team of workers communicates and collaborates. I shall list down the different infrastructure requirements for a technical IT team. Keeping in mind how digitilized our World is, they will work fine for any other distributed team.

Communication Channels

For every remote-first team, it is essential to have a communication channel. We can categorize the different communication channels by their speed. But let’s do this in the following list:

  • Email: Email is usually categorized as an official communication channel, which we can use for communication inside and outside of your organization. It is heavily asynchronous, with messages response going from minutes to days. Usually, this kind of communication is used for strategic discussions and long-term plans. That’s the reason it could be the most valuable target for an attacker.
  • Online Chat: Online chats are a faster way of exchanging messages. Usually, they are used when you need a quicker response from your peer, and there is no good time for a short, not planned call. Usually, the rule of thumb is to spend no more than 15 minutes chatting, and if the issue is not resolved, move to something faster. This one will be the second most significant target after the email for an attacker.
  • Video Conferencing: This one is the fastest. Usually, it is used to exchange a burst of already prepared information. Most of the time, the data is a tactical one, and thus this way of communication is with the lowest priority for attackers.
You can see a standard corporation infrastructure on the upper part of the diagram, where you have multiple zones and a load balancer. On the lower part is the typical small team setup, where all the data for communication is going through one cloud provider and one zone

Information Storage and Sharing

These days everything is done using information and files. But, you must store these files first and, after that, share them with your team members. Doing this using the standard communication channels is no good because there are no excellent categorization and tagging tools implemented in these systems. In short, they are not appropriately tailored for this kind of activity. That’s the reason the IT industry created a good amount of tools for solving this problem. Our small team will use them as well. So let me list them:

  • Project/Product Management System: Project management software (PMS) can help plan, organize, manage resource tools and develop resource estimates. Depending on the sophistication of the software, it can manage estimation and planning, scheduling, cost control, and budget management, resource allocation, collaboration software, communication, decision-making, quality management, time management, and documentation or administration systems. As you can see, most of the vital information for your project/product will be in this system, making it an excellent target for an attacker.
  • Cloud Storage Solution: Project management systems are outstanding in documentation storage, but they lack some of the features a full-scale cloud-based storage solution can offer. In this kind of solution, you usually store big files in a format such as video, audio, high definition graphics, etc. As such, you can leave a big part of your intellectual property lying in such cloud storage, making it a good target for an attacker.
  • Automation System: Especially in IT teams, sometimes your team will need automated jobs to happen. If you have automation specialists, know how to write scripts, you can automate a big part of your daily routes. In the case of programming teams, this is usually building, deploying, and testing procedures for a new version of your product/project. It means that you have to give access from your automation system to your programming code, for example. And this makes it an excellent target for an attacker. 

As we already discussed in the upper paragraphs, you need at least these six types of systems working and secured to have a functional remote-first team. Coming back to our knowledge of network defenses, the ideal solution for these systems is to be defended by VPN or a similar solution and expose only port 25 of the email server to support external communication. 

Unfortunately, this kind of setup will be possible only if you deploy the services in your infrastructure. In the case of cloud providers, you don’t have much control of what is exposed to the Internet and how the cloud provider takes care of your security. Plus, the infrastructure is shared between multiple organizations, and there is no guarantee that these organizations follow such strict cybersecurity rules, such as your team.

Budget

 But anyway, let’s create a budget for on-premise deployment of your infrastructure, and we shall use a VPS provider because it will be cheaper for us. A virtual private server runs its copy of an operating system (OS). Customers may have super user-level access to that active system instance, so they can install almost any software that runs on that OS. For many purposes, it is functionally equivalent to a dedicated physical server and, being software-defined, can much more easily be created and configured.

The most famous VPS providers are Amazon Web Services and Microsoft Azure. Still, there are some smaller players, such as Digital Ocean and Hetzner. As we shall do the infrastructure for a small team, we shall need a VPS with a not big pool of resources and go for the lowest price, which means CX1 instance in Hentzer. So let’s list now the different servers we shall need. All the prices are per month.

  • Email And Chat Server(3.57$): As there will be no significant demand for these two services, we can place them on the same machine.
  • VPN Server(3.57$): We shall have one machine for the VPN server, and all of the services without port 25 will be behind this VPN.
  • GitLab Server(3.57$): Gitlab is a project management/automation system. As it can become quite a hungry beast, a standalone instance is a way to go.
  • Video Conferencing Server(3.57$): One more hungry beast, it is a good idea to have it as a standalone server.
  • Cloud Storage(9.31$): This one will be a CX1 instance + an additional 100GB to store larger files. For a small team, a total of 120GB will be enough.

With a total budget of around 23.59$ per month, we achieved a pretty good level of security. Still, a determined attacker can penetrate this setup, but it will take him more time and resources. We shall use the standard VPS provider firewall. Still, if we want to achieve a higher level of security, we could add a server that will serve as a software-based firewall and IPS solution. Additionally, there are Open Source solutions for all the services types, and they will cost us 0$ per month.