My own IOT server or other option?


#1

I am getting a bit frustrated as I had an ideal local setup let down by a Raspberry Pi. Its wireless was not reliable and it had a habit of freezing.

Sensor > Nodemcu > Pi ( Python ) > webpage ( php and JavaScript )

I have been around the block a couple of times without finding a satisfactory alternative:

Sensor > Nodemcu > PC ( Python ) > webpage ( php and JavaScript ) - PC is not always on
Sensor > Nodemcu ( save data until the PC is on ) > PC ( Python ) > webpage ( php and JavaScript ) - Not very practical
Sensor > Nodemcu > NAS ( Python ) > webpage ( php and JavaScript ) - NAS is always on but the IOT software was using a lot of resources; I can’t remember exactly but it was either 98% CPU or RAM
Sensor > Nodemcu > webpage form ( php and JavaScript ) - My website is now HTTPS and I can not get a connection. Even if I can the SSL certificate changes every year and so the code will need updating.
Sensor > Nodemcu > IOT site - Works well but the data is only held for 30 days and I am not 100% happy with the webpage. I have to remember to download the data manually as I can not seem to get a way to do it automatically.

I would be happy using an alternative to the Pi if it was reliable but what?

I was thinking of setting up a IoT server on my website. Has anyone any experience?

If the worst comes to the worst I will go with the commercial Iot option. Thinking about if I forget to download the data it is a rolling 30 days; so unless I forget for more than 30 days I should not lose anything!

Edit: A concern with the NAS is it is my main backup and I did not want to risk corrupting it while setting up Python etc. As it is I had someone trying to hack into it within six weeks of setting it up!


#2

Just had a thought about rebooting the Pi if it losses connection and I have found a couple of tutorials explaining how to do it and I think I will give that a go.


#3

I am really starting to hate the Pi. Just reinstalled the OS and installed the WiFi restart script along with a touch screen and after a couple of hours I have noticed I can not get access to the Pi via wireless. It still appears on my router/hub but just times out using putty and filezilla. I could access it through the touch screen but:
The screen is another nightmare. I tried to install a touch screen keyboard a year ago and gave up as nothing worked - missing dependances, wrong versions; basically a total mess.
I could not be bothered to go through all that again and just went with the basic setup. Now I can’t access the pi as I can not get to the menu as it is off the top of the screen. This problem has been discussed since the screen was available and still not sorted.

You can see the reason I hate Linux.

EDIT: Just found a long ethernet cable and the Pi is contactable over that so it must be a WiFi problem. I suppose I could use that instead of wireless.


#5

I forwarded this topic and our messages to my friend in UK who has developed both WiFi and GPS systems. He has been working on the hardware for the past couple of years and I have developed an online result monitoring web site.

His comment:

Oh dear, I feel for him. Pretty much that’s what I started with nearly 3 years ago. It’s quite a journey. For what he’s wanting to do, LoRaWAN would be perfect.

Edit:

This is the type of temperature and humidity device posted to me from the UK. The device managed to connect to a Bangkok LoRaWAN Gateway as soon two AA batteries were fitted. The LoRaWAN Gateway relays information through to https://TheThingsNetwork.org. JSON data is then sent to our server. I have a PHP Cron job checking every minute for changes which are stored to a Mysql database. Charts and results are rendered on my website.


#6

Thank you for the interesting information @John_Betong but I think LoRaWAN is a bit OTT for what I need and will persevere with my setup for now.


#7

Just had a look at the thingsnetwork site again and I found I have a gateway about 4km away. I have also by chance found a Lora Development Board which would probably be suitable. When I looked earlier only found Arduino uno sized items at about £40. This board ( £13.99 ) has a built in temperature sensor, external arial and Lipo charger. I am a bit unsure about using the Lipo charger as the cheap one I am currently using may have damaged my battery ( which cost a lot more than the charger ) and would probably use my own sensor. There is another board on Amazon without these two items.

It still does not help with saving the data as my PC is not always on and that is why I was using the Pi. Please can you explain how you do this part “JSON data is then sent to our server”.

Edit: The LoRa chip may be a problem as only a certain one works with the Arduino library and the adverts do not say what chip is used!


#8

I have been very disappointed with my gateway connectivity assumptions. My sensor only connects when located about two metres from my nineteenth floor apartment window. Walking the full circumference of the twenty first floor of the same apartment only managed to connect to the same Gateway.

Local tests with the sensor in my shirt pocket and across town about five Kms in taxis failed to connect even though I was a lot nearer to the known gateway. Both occasions the fifth floor food court malls were visited and all building sides were tried.

I believe the majority of gateways shown on TheThingsNetwork are in “back bedrooms” and not commercial sites. Extremely long connectivity distances have been achieved, some over 100 kms, others through satellites and the Northallerton, UK gateway connectivity distance has been increased by strapping a sensor to a drone,

Line of sight is essential and also may be hindered by physical instructions such as walls, trees, buildings, etc. Some obstructions are beneficial and increase the range by acting as reflectors.

Creating our own Gateways has sometimes been necessary to ensure customers connectivity. One famous football stadium location has four gateways and relays to connect about one hundred fridge and freezer sensors. Thus was necessary to ensure connectivity with the sensors in a cellar.


#9

Many sensors have a sleep mode because continuous monitoring is not required. Good quality non rechargeable batteries can last month’s or years depending on the connectivity usage frequency.

I am not sure how the “JSON data arrives on the server”. TheThingsNetwork server sends data somehow across all platforms. No doubt they gave tried, tested and selected the easiest and most compatible methods. It is there so I check for changes and update the relevant Mysql tables.

The GPS data it is sent via our own Gateway and received on my server through $_POST parameters passed to a PHP file which inserts the data into a Mysql table. Perhaps TheThingsNetwork does something similar with the JSON data.

I will try to get my knowledgeable friend to answer your questions.


#10

I found that some of the boards I was looking at specified a range of 3-5km. I have a lot of buildings between me and the gateway and as you suspected it is probably a domestic installation:

Altitude (m): 0
Placement: indoor
Brand: Raspberry Pi DIY
Model: IMST
Antenna model: unknown

Using a Pi is also a bit worrying with the reliability problems of my Pi; is the gateway reliable? Probably best to stick with what I have; but this was an interesting conversation.
Looks like you were luck to be living where you are otherwise you would not have a connection.

My setup is solar/battery and goes to sleep after transmitting every 10min. I have found a small electronic problem as I have the solar panel > voltage regulator > battery/charger > voltage regulator > microprocessor/wireless. The voltage regulators and noise supressing capacitors all use some power which is very small but it is all the time, even when the microprocessor is sleeping.

“The GPS data it is sent via our own Gateway and received on my server through $_POST parameters passed to a PHP file which inserts the data into a Mysql table.”
I am having a problem with this part as my server is https and I think it is stopping the microprocessor saving the data into the php form. I actually get a page not found error and the library I am using is for http. If I add the SSL “finger print” it might work but my SSL certificate expires every year which means changing the code.
It must be possible as the io Adafruit site is https.


#11

As mentioned the football stadium has four Gateways each with their own RaspberryPi. Current software is C because it has "Over The Air" updates capability. LUA software was originally used but found to be inadequate.


One solution would be to create a HTTP sub-domain on your existing site. No doubt there are other ways but I think that would be the simplest because it would be completely separate from the existing .HTTPS setup.


The addition of voltage regulators, solar panels, noise suppression capacitors, etc seems to be adding unnecessary complexity and excessive power consumption. Correctly using the builtin sleep facility should negate the extra hardware. Exhaustive field tests has revealed that using good quality batteries can last over a year. Also LoRaWAN returned results include battery voltage. A screen-dump of the customised dashboard is shown. I have already been warned (with SMS and email) about the low battery voltage and curious to know how low it will go until the sensor stops sending :slight_smile:


#12

My 5V solar panel can output more than 6V and the maximum supply voltage to the charger was from memory 5V. My charger either charges the battery or supplies the microprocessor and the battery/charger output can be 4.2V and the microprocessor maximum voltage is 3.6V; hence the voltage regulators.
I also use deep sleep and the readings are transmitted every 10min and the microprocessor wakes a min before the transmission to make sure it is ready for the Pi request.

I had a problem with my PC and lost the My documents folder but from memory this setup would run for about a week without the solar panel ( 3700Ah Lipo battery ). With the solar panel it ran for probably three months until the Pi started playing up.


#13

I’m wading in to the conversation a bit late here. Been a nightmare trying to resurrect an earlier Sitepoint account. Turns out that Sitepoint Admin were not able to resurrect the account because there was some doubt as to whether me was actually me. Anyhow, this is a new account but I’ve been a Sitepoint member for 8+ years.

@Rubble We spent three years developing the AmbaSense temperature sensors (@John_Betong has posted some pictures already). We went through similar loops to you. It’s quite a learning curve. Our LoRaWAN sensors are designed and built here in the UK. They are fully LoRaWAN compliant. We’ve also developed the firmware, which functions in two scenarios: a) The Things Network (TTN) and b) As a standalone LoRaWAN network. Also developed here in the UK is our Gateway which is based upon a Raspberry Pi 3b. As mentioned, we’ve been through the loop you’re going through.

Our Raspberry Pi LoRaWAN Gateways feature two RFM95 transceivers, one acting as a listener and one acting as a ‘repeater’, forwarding data to a second Gateway. Our biggest customer installation consists of 200+ sensors, 3 Gateways and 3 Repeaters. These are installed at a UK Premiership football club and cover the whole of the stadium. We have redundancy in the Gateways for backup purposes. Each Gateway is mains powered - we would NOT consider battery power for the Pi.

Sensors are powered by two AA Alkaline cells and so far, real-world batteries have lasted 1 year and 3 months with 84% power remaining. The important part with sensor powering is to code the MCU and temperature sensor so that they are ASLEEP for most of their lives. Our MCU wakes itself up every 40~ minutes to take a reading. Quiescent current is about 10 μA.

Happy to help out if we can.


#14

Thank you for the information @ambasense and I have looked at your website. LoRaWAN looks a bit more powerful than I would need as I am only transmitting about 10m. I had a look at the LoRaWAN network and my nearest gateway was about 4km away, through a housing estate and I do not think I would get a signal. I would also need a different microprocessor/wireless for the sensor although that is not a problem.

Since first starting this thread I have done some more testing:
I currently have a sensor in the greenhouse that is sending the data to the IO.Adafruit website. This works well but I do not particularly like the Adafruit dashboard as there are not many options.
I have resurrected the sensor pi setup and it failed within the first two hours even after a complete reinstall of the pi. I restarted it and it has been running in the study with a sensor about 3m away since the 28th January - this is my preferred setup.
I now have a non https subdomain and can send the data from the sensor to a webform and save it into a database. Posibly the method I will use now.
I have also sent data from the pi to my website via ssh but I can not yet do that from the sensor. It was suggested I used a pi zero and I could have that in the greenhouse with the sensor and use ssh to upload the data to my website. I do not think I can put the pi to sleep without an extra piece or hardware.

Anyway I have a few things to think about now that work.


#15

I had thought about this briefly and thought about using a traditional relay to power the sensor when the microprocessor woke up but thought that would draw more power than the sensor so dismissed it. But if I used a transistor/MOSFET ( I always forget about this method ) the current should be less; although more than the sensor current at any particular time. But as the microprocessor is asleep for 9.5min and only awake for less than a minute it could work out less.
Anyway something to check out tomorrow; power consumption is not so important now as I have picked up a couple of second hand four cell Lipo battery packs.