{"id":85,"date":"2014-03-03T22:48:13","date_gmt":"2014-03-03T22:48:13","guid":{"rendered":"http:\/\/www.flat-planet.net\/?page_id=85"},"modified":"2014-03-17T20:52:51","modified_gmt":"2014-03-17T20:52:51","slug":"arduino-networked-datalogger","status":"publish","type":"page","link":"https:\/\/flat-planet.net\/?page_id=85","title":{"rendered":"Arduino Networked Datalogger"},"content":{"rendered":"<p>This little project is a combination of a few of the example code, specifically UDP NTP Client and SD-Datalogger. What I&#8217;m aiming for is that I can log the values from a bunch of sensors periodically into a log file, and periodically upload the file to a server for graphing\/processing.<\/p>\n<p>Some obvious questions that would occur as to why and do it this way rather then pushing each sensor reading up to the server\/website:<\/p>\n<ol>\n<li>Real-time sensor information is not required, but high-resolution of data may be<\/li>\n<li>I don&#8217;t want to assume that network connectivity is always available<\/li>\n<li>Constantly pushing tiny chunks of data to a server is a bad idea if you are using mobile (2G\/3G) networks &#8211; from a network perspective this results in high signalling load relative to the data volume<\/li>\n<li>Battery life &#8211; aiming for low cost, low power sensor platform &#8211; I suspect that the WiFi Shield\/GSM shield is a bit thirsty\n<ul>\n<li>Need to see if that can be turned on and off on a timer<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n<p>I&#8217;m planning to update this page with different sections and notes as I continue to experiment, and blog posts for specific problems seen. Also thinking of adding this to github for version control, etc.<\/p>\n<h2>Maintaining &#8216;true&#8217; time without an RTC<\/h2>\n<p>Building a data logger without an <a title=\"RTC Shield\" href=\"http:\/\/www.adafruit.com\/products\/243\">RTC Shield<\/a> poses an interesting challenge since, for me, a key thing is to log the data with a timestamp. My thinking is to use the UDP NTP module to get the <a href=\"http:\/\/en.wikipedia.org\/wiki\/Unix_time\">epoch time<\/a> and then try and maintain that time internally as best I can.<\/p>\n<p>Given that I don&#8217;t want the app to be chatty in networking terms, I essentially have to maintain an internal count of the time e.g. increment my epoch time counter every 1000msec. Obviously this has problems in terms of accuracy since it relies on non-blocking calls in the main loop, and short loop execution time to ensure that we can update every 1sec.<\/p>\n<p>Since this is a poor man&#8217;s RTC there will be some drift, but if I assume we get network connectivity periodically (for data log upload) then we can resync via <a href=\"http:\/\/en.wikipedia.org\/wiki\/Network_Time_Protocol\">NTP<\/a>.<\/p>\n<pre>\u00a0\u00a0timer_3\u00a0-=\u00a0deltaMillis;\r\n\u00a0\u00a0<span style=\"color: #cc6600;\">if<\/span>( timer_3 &lt;= 0 ){\r\n\u00a0\u00a0\u00a0\u00a0timer_3\u00a0+=\u00a0TIMER_INTERVAL_3; \/\/ This is 30mins\r\n\u00a0\u00a0\u00a0 delta_epoch\u00a0=\u00a0e_time;\r\n\u00a0\u00a0\u00a0\u00a0e_time\u00a0=\u00a0get_epoch_time();\r\n\u00a0\u00a0\u00a0\u00a0<span style=\"color: #cc6600;\"><b>Serial<\/b><\/span>.<span style=\"color: #cc6600;\">print<\/span>(<span style=\"color: #006699;\">\"Clock drift: \"<\/span>);\r\n\u00a0\u00a0\u00a0\u00a0<span style=\"color: #cc6600;\"><b>Serial<\/b><\/span>.<span style=\"color: #cc6600;\">println<\/span>( (e_time - delta_epoch) ); \/\/ negative drift will be a huge number\r\n\u00a0 }<\/pre>\n<p>In a very simple setup the clock drift has been averaging around +1 second every 30mins which I consider acceptable.<br \/>\n<code><br \/>\nInitializing SD card...card initialized.<br \/>\nClock drift: 2<br \/>\nClock drift: 1<br \/>\nClock drift: 1<br \/>\nClock drift: 1<br \/>\nClock drift: 1<br \/>\nClock drift: 1<br \/>\nClock drift: 1<br \/>\nClock drift: 1<br \/>\nClock drift: 0<br \/>\nClock drift: 1<br \/>\nClock drift: 1<br \/>\nClock drift: 1<br \/>\nClock drift: 1<\/code><\/p>\n<h2>Sensor Charts<\/h2>\n<p>I now have a very basic system up and running with 2 sensors feeding information to a website every 60 seconds and RRD graphing. Those charts are <a href=\"http:\/\/www.flat-planet.net\/?page_id=110\">here<\/a>.<\/p>\n<p>To stop the whole world pushing data into this page I&#8217;ve got some basic security based around HTTP Header enrichment and shared secret validation &#8211; check out <a href=\"http:\/\/www.flat-planet.net\/?p=100\">this post<\/a> for the details.<\/p>\n<h2>Ethernet Shield Issues<\/h2>\n<p>I&#8217;ve had a number of issues with the &#8216;standard&#8217; Ethernet shield dying on me \ud83d\ude41 This of course makes a network logged system unusable since once it goes down, it stays down even though the rest of the program is working perfectly as far as I and the serial monitor can tell. This seems to be a widely known issue with the W5100 Ethernet chip. I&#8217;ve added some code to try and get the Arduino to get itself back on track in the event of issues, but while it has helped it seems that I will have to build in a software triggered hardware reset.<\/p>\n<pre>\u00a0\u00a0<span style=\"color: #7e7e7e;\">\/\/ No need for any delay loops as the Ethernet.begin seems to go into an interrup type rechecking<\/span>\r\n\u00a0\u00a0<span style=\"color: #7e7e7e;\">\/\/ for about 60seconds. If the connection comes up during that window we seem to complete as soon<\/span>\r\n\u00a0\u00a0<span style=\"color: #7e7e7e;\">\/\/ as link diagnostics are happy<\/span>\r\n\u00a0\u00a0<span style=\"color: #cc6600;\">while<\/span> ( <span style=\"color: #cc6600;\">Ethernet<\/span>.<span style=\"color: #cc6600;\">begin<\/span>(mac) == 0){\r\n\u00a0\u00a0\u00a0\u00a0<span style=\"color: #cc6600;\"><b>Serial<\/b><\/span>.<span style=\"color: #cc6600;\">print<\/span>(<span style=\"color: #006699;\">\"DHCP Failure - No IP. Reset counter: \"<\/span>);\r\n\u00a0\u00a0\u00a0\u00a0resetCounter++;\r\n\u00a0\u00a0\u00a0\u00a0<span style=\"color: #cc6600;\"><b>Serial<\/b><\/span>.<span style=\"color: #cc6600;\">println<\/span>(resetCounter);\r\n\u00a0\u00a0\u00a0\u00a0<span style=\"color: #cc6600;\">if<\/span> (resetCounter &gt;= 6 ){\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<span style=\"color: #cc6600;\"><b>Serial<\/b><\/span>.<span style=\"color: #cc6600;\">println<\/span>(<span style=\"color: #006699;\">\"Initiating software reboot...\"<\/span>);\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<span style=\"color: #cc6600;\">delay<\/span>(1000); \r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0software_Reboot();\u00a0\r\n\u00a0\u00a0\u00a0\u00a0}\r\n\u00a0\u00a0\u00a0\u00a0<span style=\"color: #cc6600;\">if<\/span> (resetCounter &gt;=5 ) {resetEthernetShield();}\r\n\u00a0\u00a0}\r\n\r\n\u00a0\u00a0<span style=\"color: #cc6600;\">delay<\/span>(1000); <span style=\"color: #7e7e7e;\">\/\/ give the shoed a second to initialise<\/span><\/pre>\n<p>Everytime there is a failure the resetCounter is incremented. The first 5 times, the resetEthernetShield() function is called (below), the 6th and final time we trigger the watchdog timer and reset the board (though this doesn&#8217;t seem to always reset the W5100 chipset unfortunately.<\/p>\n<pre><span style=\"color: #7e7e7e;\">\/\/\u00a0Reboot\u00a0the\u00a0whole\u00a0thing\u00a0-\u00a0in\u00a0theory.\u00a0<\/span>\r\n<span style=\"color: #7e7e7e;\">\/\/\u00a0An\u00a0actual\u00a0hardware\u00a0interrupt\u00a0to\u00a0power\u00a0would\u00a0be\u00a0better<\/span>\r\n<span style=\"color: #cc6600;\">void<\/span> software_Reboot(){\r\n\r\n\u00a0\u00a0wdt_enable(WDTO_15MS);\r\n\u00a0\u00a0<span style=\"color: #cc6600;\">while<\/span>(1){  }\r\n}\r\n\r\n<span style=\"color: #7e7e7e;\">\/\/\u00a0Soft\u00a0shield\u00a0reset\u00a0-\u00a0no\u00a0idea\u00a0if\u00a0this\u00a0helps\u00a0much<\/span>\r\n<span style=\"color: #cc6600;\">void<\/span> resetEthernetShield(){\r\n\r\n\u00a0\u00a0<span style=\"color: #cc6600;\"><b>Serial<\/b><\/span>.<span style=\"color: #cc6600;\">println<\/span>(<span style=\"color: #006699;\">\"Resetting Ethernet Shield...\"<\/span>);   \r\n\r\n\u00a0\u00a0client.<span style=\"color: #cc6600;\">stop<\/span>();\r\n\u00a0\u00a0<span style=\"color: #cc6600;\">delay<\/span>(1000);\r\n\r\n\u00a0\u00a0<span style=\"color: #cc6600;\">Ethernet<\/span>.<span style=\"color: #cc6600;\">begin<\/span>(mac);\r\n\u00a0\u00a0<span style=\"color: #cc6600;\">delay<\/span>(1000);\r\n}<\/pre>\n<p>Based on the performance of this code, a software triggered power interrupt to the Ethernet shield is going to be required as the final application would be &#8216;off-site&#8217; so it must be able to recover fast and on it&#8217;s own.<\/p>\n<p><a href=\"http:\/\/www.flat-planet.net\/wp-content\/uploads\/2014\/03\/temperature_resetAndRecovery.jpg\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-132\" alt=\"temperature_resetAndRecovery\" src=\"http:\/\/www.flat-planet.net\/wp-content\/uploads\/2014\/03\/temperature_resetAndRecovery.jpg\" width=\"747\" height=\"494\" \/><\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>This little project is a combination of a few of the example code, specifically UDP NTP Client and SD-Datalogger. What I&#8217;m aiming for is that I can log the values from a bunch of sensors periodically into a log file, and periodically upload the file to a server for graphing\/processing. Some obvious questions that would [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"parent":49,"menu_order":0,"comment_status":"open","ping_status":"open","template":"","meta":{"footnotes":""},"class_list":["post-85","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/flat-planet.net\/index.php?rest_route=\/wp\/v2\/pages\/85","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/flat-planet.net\/index.php?rest_route=\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/flat-planet.net\/index.php?rest_route=\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/flat-planet.net\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/flat-planet.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=85"}],"version-history":[{"count":6,"href":"https:\/\/flat-planet.net\/index.php?rest_route=\/wp\/v2\/pages\/85\/revisions"}],"predecessor-version":[{"id":133,"href":"https:\/\/flat-planet.net\/index.php?rest_route=\/wp\/v2\/pages\/85\/revisions\/133"}],"up":[{"embeddable":true,"href":"https:\/\/flat-planet.net\/index.php?rest_route=\/wp\/v2\/pages\/49"}],"wp:attachment":[{"href":"https:\/\/flat-planet.net\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=85"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}