tag:blogger.com,1999:blog-13845516535066043562024-02-19T04:01:01.518+01:00LC BlogLukáš Čenovskýhttp://www.blogger.com/profile/14627955690950344876noreply@blogger.comBlogger42125tag:blogger.com,1999:blog-1384551653506604356.post-82124461378639736722023-09-05T23:42:00.003+02:002023-09-05T23:54:02.737+02:00Monitor solar heating with ebusd<p>I have Wolf flat-plate collector on my roof to heat water in the bottom of the warm water tank. There is simple monitoring (Wolf BM Solar) which shows whether heating is on and temperatures on the collector and in the tank. I wanted to see this data in my Home Assistant.</p>
<p>Wolf does not support OpenTherm but eBUS. After short searching on internet, I've found <a href="https://adapter.ebusd.eu/index.en.html" target="_blank">eBUS adapter </a>which can monitor/control eBUS devices (mainly boilers). Delivery took about 2 months but finally I have it now.</p>
<p><a href="https://adapter.ebusd.eu/v5/steps.en.html" target="_blank">First steps with the adapter</a> are well described. Yet I had some issues:</p>
<ul style="text-align: left;">
<li>flashing and accessing web interface (<b>easi</b>) was fine</li>
<li>I've struggled with <b>easi</b> authentication - after entering credentials, accessing it from the same browser still worked. Only accessing it from another browser timed out and I don't see a way how to enter credentials (passing them in the url does not work). I will dig deeper into it later. UPDATE: it's fixed by <a href="https://adapter.ebusd.eu/v5/ChangeLog#20230903" target="_blank">2023-09-03</a> firmware.</li>
<li>I run <b>ebusd</b> on my Turris router in docker. For some reason, which I don't remember now, all my docker containers had forbidden access to internet. HomeAssistant is fine with it but <b>ebusd</b> needs to download configuration files and it took me a while to find out why requests for configuration were failing.</li>
</ul>
<p>The biggest issue was with configuration. My Wolf is not supported out of the box. Following <a href="https://github.com/john30/ebusd/blob/master/contrib/docker/README.md#using-a-network-device" target="_blank">docker readme</a>, I have run the following command:</p>
<pre>
root@turris:~# docker run --rm -it -p 8888 john30/ebusd --scanconfig -d ens:192.168.1.87:9999 --latency=20
2023-09-03 18:07:50.453 [main notice] ebusd 23.2.23.2 started with auto scan on device: 192.168.1.87:9999, enhanced
2023-09-03 18:07:51.734 [bus notice] device status: resetting
2023-09-03 18:07:51.735 [bus notice] bus started with own address 31/36
2023-09-03 18:07:51.740 [bus notice] signal acquired
2023-09-03 18:07:52.240 [bus notice] device status: reset, supports info
2023-09-03 18:07:52.355 [bus notice] device status: extra info: firmware 1.1[380f].1[380f], jumpers 0x0f
2023-09-03 18:07:58.939 [bus notice] new master 71, master count 2
2023-09-03 18:07:58.939 [update notice] received unknown BC cmd: 71fe50180e0080008000800080008000800000
2023-09-03 18:08:02.009 [bus notice] scan 76: ;Kromschroeder; ;0227;-
2023-09-03 18:08:02.009 [update notice] store 76 ident: done
2023-09-03 18:08:02.009 [update notice] sent scan-read scan.76 QQ=31: Kromschroeder; ;0227;-
2023-09-03 18:08:02.009 [bus notice] scan 76: ;Kromschroeder; ;0227;-
2023-09-03 18:08:02.219 [main error] HTTP failure: receive error: 404 Not Found
2023-09-03 18:08:02.219 [main error] unable to load scan config 76: list files in kromschroeder ERR: element not found
2023-09-03 18:08:02.219 [main error] scan config 76: ERR: element not found
2023-09-03 18:08:13.921 [update notice] received unknown BC cmd: 71fe50171000011a016b0200800080008000800080
2023-09-03 18:08:28.879 [update notice] received unknown BC cmd: 71fe50180e0080008000800080008000800000
</pre>
<p>I connected to docker shell and get more info:</p>
<pre>
root@600974f7960e:/# ebusctl i
version: ebusd 23.2.23.2
device: 192.168.1.87:9999, enhanced, firmware 1.1[380f].1[380f]
signal: acquired
symbol rate: 20
max symbol rate: 47
min arbitration micros: 4
max arbitration micros: 4
min symbol latency: 17
max symbol latency: 19
scan: finished
reconnects: 0
masters: 2
messages: 12
conditional: 0
poll: 0
update: 4
address 31: master #8, ebusd
address 36: slave #8, ebusd
address 71: master #9
address 76: slave #9, scanned "MF=Kromschroeder;ID= ;SW=0227;HW=-"
</pre>
<p>Using <a href="https://github.com/john30/ebusd/wiki/HowTos" target="_blank">HowTos</a> and <a href="https://github.com/john30/ebusd/wiki/4.1.-Message-definition" target="_blank">Message definition</a>, I was able to decode above unknown messages:</p>
<pre>
2023-09-03 18:08:13.921 [update notice] received unknown BC cmd: 71fe50171000011a016b0200800080008000800080
2023-09-03 18:08:28.879 [update notice] received unknown BC cmd: 71fe50180e0080008000800080008000800000
root@7e8bde847891:/# ebusctl grab result
71fe50171000010301580200800080008000800080 = 2
71fe50180e0080008000800080008000800000 = 3
</pre>
<p>There is a master device (id <code>71</code>) - that's the solar heating. Then there is a slave device (id <code>76</code>) which is Wolf BM Solar module (it displays data from master). The master broadcasts (<code>BC</code> in log or <code>fe</code> in message) messages. <code>PBSB</code> of messages are <code>5017</code> and <code>5018</code>.</p>
<p>I've searched through <a href="https://github.com/john30/ebusd-configuration" target="_blank">ebusd-configuration</a> repo for <code>PBSB</code> codes and found them in <a href="https://github.com/john30/ebusd-configuration/blob/master/ebusd-2.x.x/de/wolf/50.csv" target="_blank">ebusd-2.x.x/de/wolf/50.csv</a>.</p>
<p>To be able to use updated configs, I've created <code>ebusd_config</code> folder on turris that is shared to ebusd docker container. Based on <code>unable to load scan config 76: list files in kromschroeder</code> error, I've created <code>kromschroeder</code> subfolder there where I</p>
<ul style="text-align: left;">
<li>copied <code>ebusd-2.x.x/de/_templates.csv</code> to <code>ebusd_config/kromschroeder/_templates.csv</code></li>
<li>copied <code>ebusd-2.x.x/de/wolf/50.csv</code> to <code>ebusd_config/kromschroeder/76.csv</code></li>
<li>copied <code>ebusd-2.x.x/de/wolf/broadcast.csv</code> to <code>ebusd_config/kromschroeder/broadcast.csv</code> and added 2 lines with missing <code>PBSB</code> codes (changing message type to broadcast <code>b</code>):<br/>
<pre>
# type (r;w;u;1-9),class,name,comment,QQ,ZZ,PBSB,ID,field1,part (m;s),type / templates,divider / values,unit,comment,field2,part (m;s),type / templates,divider / values,unit,comment,field3,part (m;s),type / templates,divider / values,unit,comment,field4,part (m;s),type / templates,divider / values,unit,comment,field5,part (m;s),type / templates,divider / values,unit,comment,field6,part (m;s),type / templates,divider / values,unit,comment,field7,part (m;s),type / templates,divider / values,unit,comment
*b,broadcast,,,,FE,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
b,,betrd,Betriebsdaten des Feuerungsautomaten an den Regler,,,0503,01,status,,UCH,,,Feuerungsautomatstatus,zustand,,HEX,,,Feuerungsautomatzustand,stellgrad,,percent0,,,StellgradKesselleistung (Wolf Gastherme Ersatzwert: FF),kesseltemp,,temp1,,,Kesseltemperatur ,ruecklauftemp,,temp0,,,Rücklauftemperatur ,boilertemp,,temp0,,,Boilertemperatur ,aussentemp,,outsidetemp,,,
b,,sollw,Sollwertübertragung des Reglers an andere Regler,,,0800,,kesselsolltemp,,temp2,,,Kesselsollwert,aussentemp,,temp2,,,Außentemperatur,leistungszwang,,D1B,,%,Leistungszwang,status,,HEX,,,Status,brauchwassersolltemp,,temp2,,,Brauchwassersollwert,,,,,,,,,,,,
b,,ertraege,,,,5018,,leistung,,D2B,,,Solarleistung,ertraglow,,UCH,,,ErtragTagLow,ertraghigh,,UCH,,,ErtragTagHigh,ertragsummelow,,UCH,,,SummeErtragLow,summertraghigh,,UCH,,,SummeErtragHigh,ertragsumme,,UCH,,,SummeErtragM,,,,,,
b,,temp,,,,5017,,pumpe,,UCH,0xbc=off;0xbd=on,,SolarPumpe,,,IGN:1,,,,kollektortemp,,temp,,,Kollektortemperatur,wwsolartemp,,temp,,,WW Solartemperatur,,,,,,,,,,,,,,,,,,
</pre>
</li>
</ul>
<p>I ran the docker again and success:</p>
<pre>
root@turris:~# docker run --rm -it -p 8888 -v "/srv/ebusd_config:/mnt:ro" john30/ebusd --scanconfig --configpath=/mnt -d ens:192.168.1.87:9999 --latency=20
2023-09-03 21:03:05.609 [main notice] ebusd 23.2.23.2 started with auto scan on device: 192.168.1.87:9999, enhanced
2023-09-03 21:03:05.853 [bus notice] device status: resetting
2023-09-03 21:03:05.854 [bus notice] bus started with own address 31/36
2023-09-03 21:03:05.873 [bus notice] signal acquired
2023-09-03 21:03:06.305 [bus notice] device status: reset, supports info
2023-09-03 21:03:06.473 [bus notice] device status: extra info: firmware 1.1[380f].1[380f], jumpers 0x0f
2023-09-03 21:03:15.326 [bus notice] new master 71, master count 2
2023-09-03 21:03:15.326 [update notice] received unknown BC cmd: 71fe5017100001e000430200800080008000800080
2023-09-03 21:03:16.113 [bus notice] scan 76: ;Kromschroeder; ;0227;-
2023-09-03 21:03:16.113 [update notice] store 76 ident: done
2023-09-03 21:03:16.113 [update notice] sent scan-read scan.76 QQ=31: Kromschroeder; ;0227;-
2023-09-03 21:03:16.113 [bus notice] scan 76: ;Kromschroeder; ;0227;-
2023-09-03 21:03:16.115 [main notice] read common config file kromschroeder/broadcast.csv
2023-09-03 21:03:16.116 [main notice] read scan config file kromschroeder/76.csv for ID "", SW0227, HW65535
2023-09-03 21:03:16.116 [main notice] found messages: 8 (0 conditional on 0 conditions, 0 poll, 4 update)
2023-09-03 21:03:30.288 [update notice] received update-read broadcast ertraege QQ=71: -;0;128;0;128;0
2023-09-03 21:03:45.274 [update notice] received update-read broadcast temp QQ=71: 0;14.00;36.19
</pre>
<p>Data from master broadcast (<code>temp QQ=71: 0;14.00;36.19</code>) tells me that:</p>
<ul style="text-align: left;">
<li><code>0</code> - pump is not running</li>
<li><code>14.00</code> - collector temperature</li>
<li><code>36.19</code> - warm water tank temperature</li>
</ul>
<p>I'm not interested in <code>ertraege</code> (yield of the solar) - gathering is turned off anyway.</p>
<p>The final command to run ebusd in the background has to use environment variables to set the parameters:</p>
<pre>docker run --name ebusd -d -p 9090:9090 -v "/srv/ebusd_config:/mnt:ro" -e EBUSD_DEVICE=ens:192.168.1.87:9999 -e EBUSD_SCANCONFIG= -e EBUSD_CONFIGPATH=/mnt -e EBUSD_LATENCY=20 -e EBUSD_HTTPPORT=9090 john30/ebusd</pre>
<p>There are <a href="https://github.com/john30/ebusd/wiki/7.-Integrations#integration-with-home-assistant" target="_blank">several options</a> how to get data from <b>ebusd</b> to Home Assistant. As I don't use MQTT and I need just 2 values, I've decided to use simple polling via http (that's why <code>EBUSD_HTTPPORT</code> parameter in above command). I've tried to secure it by <code>-e EBUSD_READONLY=</code> but that resulted in failures of reading the configuration (which is actually expected as I have found later).</p>
<pre>
rest:
- resource: http://192.168.1.1:9090/data/broadcast/temp
scan_interval: 15
binary_sensor:
- name: Solar - running
unique_id: solar_ohrev_id
device_class: running
value_template: "{{ value_json['broadcast']['messages']['temp']['fields']['pumpe']['value']|int == 1 }}"
sensor:
- name: Solar - collector temperature
unique_id: solar_kolektor_id
state_class: measurement
device_class: temperature
value_template: "{{ value_json['broadcast']['messages']['temp']['fields']['kollektortemp']['value'] }}"
unit_of_measurement: °C
- name: Solar - tank temperature
unique_id: solar_teplota_vody_id
state_class: measurement
device_class: temperature
value_template: "{{ value_json['broadcast']['messages']['temp']['fields']['wwsolartemp']['value'] }}"
unit_of_measurement: °C
</pre>
<p>This is graph from today's sunny weather. It's nicely visible how the full tank was heated to over 60°C.</p>
<ul style="text-align: left;">
<li><code>red line</code> - collector's temperature</li>
<li><code>blue line</code> - bottom tank temperature</li>
<li><code>orange line</code> - top tank temperature</li>
</ul>
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgGj9seKQ3NqIHFQo7zBXfnT18zxPEHKC4YA9N2HS09mpcZA_OuBm4pZJNwxr42jGpHk1iQ7qXGMoIe8vveMOJQ4srw9K3WPNB2S76QAjzHaYCy3I1UADi1wqRqEEmy28OkQY6ZdLmczlTbnQGVxt5gWJ6t-PYuFavEQUOjkXlz3XtjAzvNB-B8o1swlW0/s436/Snipaste_2023-09-05_23-34-12.png" style="display: block; padding: 1em 0; text-align: center; clear: left; float: left;"><img alt="" border="0" data-original-height="306" data-original-width="436" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgGj9seKQ3NqIHFQo7zBXfnT18zxPEHKC4YA9N2HS09mpcZA_OuBm4pZJNwxr42jGpHk1iQ7qXGMoIe8vveMOJQ4srw9K3WPNB2S76QAjzHaYCy3I1UADi1wqRqEEmy28OkQY6ZdLmczlTbnQGVxt5gWJ6t-PYuFavEQUOjkXlz3XtjAzvNB-B8o1swlW0/s320/Snipaste_2023-09-05_23-34-12.png"/></a></div>Lukáš Čenovskýhttp://www.blogger.com/profile/14627955690950344876noreply@blogger.com0tag:blogger.com,1999:blog-1384551653506604356.post-62218289257604741812023-01-08T23:27:00.023+01:002023-09-05T23:48:01.410+02:00Quest for lower electricity consumption - What does burn the heat?<p>Equipped with long term data, I started performing tests. As described <a href="https://gui-at.blogspot.com/2022/12/quest-for-lower-electricity-consumption_30.html" target="_blank">previously</a>, my home setup is not that simple.</p><p></p><div class="separator" style="clear: both; text-align: left;"><div class="separator" style="clear: both; text-align: center;"><div class="separator" style="clear: both; text-align: center;"><div class="separator" style="clear: both; text-align: left;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEiOqXKuJIWQk_U71LLIbozjDO8yx2URRmqtA0hFDLliqQU6W6--gfM7We1AlZEOKdTx_gsT9p2Vtone1vLWPCIRnGqfRhJvSlv7R5BVOdI4d0ykOp1kkdjuQrONvh-PTBYtvTmz73x_aema8A-m_sC-e8QMAFLkj7Io4abIP2TNPr4sfnFFaXHxpT1R" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="336" data-original-width="764" height="282" src="https://blogger.googleusercontent.com/img/a/AVvXsEiOqXKuJIWQk_U71LLIbozjDO8yx2URRmqtA0hFDLliqQU6W6--gfM7We1AlZEOKdTx_gsT9p2Vtone1vLWPCIRnGqfRhJvSlv7R5BVOdI4d0ykOp1kkdjuQrONvh-PTBYtvTmz73x_aema8A-m_sC-e8QMAFLkj7Io4abIP2TNPr4sfnFFaXHxpT1R=w640-h282" width="640" /></a></div></div></div></div><p></p><p>Heating pipe for warm water tank is not separated from central heating pipes by three-way valve. When heating up the tank, the pump for tank circle runs and the other two are off. Vice versa when heating the central heating branches.</p><p>This is the usual temperature graph for a night (that means no warm water concumption):</p><p></p><div class="separator" style="clear: both; text-align: left;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEhqQPiayCK60W0lD8mkPoFxtr9A2iJRBbZ5btoA9yOPqg26xpr1cncjC0ze6yfXRWo9lzYLh0jK9wmjq3BCQhQBZabOFFtAFxDkY6NU7irfKLbjy2-wT9WMtiahvKIMcgTbeBQiflAMC4g6LtE0QmYDhA2mNLqy1ER68cN8ij6JFTG75pj3vEqdsuv7" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="303" data-original-width="498" height="244" src="https://blogger.googleusercontent.com/img/a/AVvXsEhqQPiayCK60W0lD8mkPoFxtr9A2iJRBbZ5btoA9yOPqg26xpr1cncjC0ze6yfXRWo9lzYLh0jK9wmjq3BCQhQBZabOFFtAFxDkY6NU7irfKLbjy2-wT9WMtiahvKIMcgTbeBQiflAMC4g6LtE0QmYDhA2mNLqy1ER68cN8ij6JFTG75pj3vEqdsuv7=w400-h244" width="400" /></a></div><p>From 65°C to 50°C in approximately 2,5 hours. The reason for not fully heated tank around 1:00 is there more expensive electric tariff between 1:00 - 2:00 and the boiler is turned off.</p><p></p><div style="text-align: left;"><b>Hypothesis</b></div><div style="text-align: left;">My first idea was the water in tank is cooled by water from the central heating. So I closed valves for a night and checked the graph.</div><p></p><div class="separator" style="clear: both; text-align: left;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEj4ZCQQF0WV2k2OZVd0zULwmEHtmYrjhHUAMnU4yYjmsTOS0dr4Rdt2RHLraeA4uSlApF-3NOvkdflZbHdQigqZRx73N1TYsLVyHfPspDiic6JWZKXsRatB894H-QATJ-e1AeGI96W1GjU1z9JDgyG1B2KODcGlFh0ISwB5py8BHMSBLB9GPCtGSlmu" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="303" data-original-width="498" height="244" src="https://blogger.googleusercontent.com/img/a/AVvXsEj4ZCQQF0WV2k2OZVd0zULwmEHtmYrjhHUAMnU4yYjmsTOS0dr4Rdt2RHLraeA4uSlApF-3NOvkdflZbHdQigqZRx73N1TYsLVyHfPspDiic6JWZKXsRatB894H-QATJ-e1AeGI96W1GjU1z9JDgyG1B2KODcGlFh0ISwB5py8BHMSBLB9GPCtGSlmu=w400-h244" width="400" /></a></div><br />The cooling rate from 65°C to 45°C is the same as for any other night, so cooler water for the central heating is not the culprit. Why the tank cannot keep hot water longer was a mystery...<p></p><div style="text-align: left;"><b>Culprit</b></div><div style="text-align: left;">Finding the real culprit required having a beer with neighbors in a pub. More head, the greater knowledge and soon we had a hypothesis that the loss of heat is caused by natural circulation. My warm water tank is in the basement and warm water pipes go through the house and returns to the tank. As pipes are relatively wide, they allow natural circulation. When I stopped the circulation, the graph looked much better. The loss was less than 15°C in 8 hours.</div><div><br /></div><div><div class="separator" style="clear: both; text-align: left;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEjvwX2D-nJVA94ATjreELLCD2bCzaGdFoEQTXvjXDbc5TWVaPBeqm1zqIF8sxJcYi5nFf1MnTwMs96_B8OHBj8P9JB-QiTsL20BMEzZmraAFFy3lN--3NqMxCQFDv4DBiM6t2y4wTs24Tq7eP-1r_kh535isgNUCCumL-HWeGwJz_Xv8g3zylEYM8SP" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="303" data-original-width="498" height="244" src="https://blogger.googleusercontent.com/img/a/AVvXsEjvwX2D-nJVA94ATjreELLCD2bCzaGdFoEQTXvjXDbc5TWVaPBeqm1zqIF8sxJcYi5nFf1MnTwMs96_B8OHBj8P9JB-QiTsL20BMEzZmraAFFy3lN--3NqMxCQFDv4DBiM6t2y4wTs24Tq7eP-1r_kh535isgNUCCumL-HWeGwJz_Xv8g3zylEYM8SP=w400-h244" width="400" /></a></div><br />Now I just have to figure out what to do. With no circulation, it takes quite a long time to get warm water from the tap. With circulation, the cost of heating is high... </div><div><br /></div><div>My current idea is to use electric valve and open the circulation just for a while, probably speeding it up with a pipe.</div><div><br /></div><div><br /></div><div><b>Edit:</b></div><div>The real culprit was found when new tank was installed - the reverse flap was faulty and caused natural circulation. Once fixed, the warm water can circulate only in one direction again and there is no natural circulation anymore.</div><br/>Lukáš Čenovskýhttp://www.blogger.com/profile/14627955690950344876noreply@blogger.com0tag:blogger.com,1999:blog-1384551653506604356.post-47887064705507013622023-01-03T21:47:00.021+01:002023-01-03T23:02:35.405+01:00Quest for lower electricity consumption - Investigate Arduino Crash<p>As I wrote <a href="https://gui-at.blogspot.com/2023/01/quest-for-lower-electricity-consumption.html" target="_blank">last time</a>, my joy did not last long because the data flew only for 1-2 hours and then it stopped. I connected WeMos to my laptop and boiler and kept it running while logging the progress to serial monitor. Eventually, it crashed (<a href="https://github.com/CendaL/thermona-opentherm/blob/1a3a9a79c6e7f4665d201dea4eeba2803d476cf5/crash/raw.output.txt" target="_blank">full log</a>):</p>
<pre><code>
11:21:19.141 -> --------------- CUT HERE FOR EXCEPTION DECODER ---------------
11:21:19.141 ->
11:21:19.141 -> Exception (0):
11:21:19.141 -> epc1=0x402010f4 epc2=0x00000000 epc3=0x00000000 excvaddr=0x00000000 depc=0x00000000
11:21:19.141 ->
11:21:19.141 -> >>>stack>>>
11:21:19.141 ->
11:21:19.141 -> ctx: sys
11:21:19.141 -> sp: 3fffebb0 end: 3fffffb0 offset: 0190
11:21:19.141 -> 3fffed40: 3ffefa44 3ffee748 3ffee740 401001e2
11:21:19.141 -> 3fffed50: 3ffe0000 3ffeec30 3ffee740 4010041b
...
11:21:20.500 -> 3fffff90: 3fffdad0 00000000 3ffee824 401004dd
11:21:20.500 -> 3fffffa0: 3fffdad0 00000000 3ffee824 40206906
11:21:20.500 -> <<<stack -="" 11:21:20.500="">
11:21:20.500 -> --------------- CUT HERE FOR EXCEPTION DECODER ---------------
11:21:20.500 ->
11:21:20.500 -> ets Jan 8 2013,rst cause:2, boot mode:(3,6)
11:21:20.500 ->
11:21:20.500 -> load 0x4010f000, len 3460, room 16
11:21:20.500 -> tail 4
11:21:20.500 -> chksum 0xcc
11:21:20.500 -> load 0x3fff20b8, len 40, room 4
11:21:20.500 -> tail 4
11:21:20.500 -> chksum 0xc9
11:21:20.547 -> csum 0xc9
11:21:20.547 -> v000604e0
11:21:20.547 -> ~ld
</stack></code></pre>
<p>The problem was that although WeMos did somehow restart, it did not pass the step that connects it again to wifi and stuck there. I had to find why it crashed.</p><p><a href="https://arduino-esp8266.readthedocs.io/en/latest/faq/a02-my-esp-crashes.html" target="_blank">ESP documentation</a> shows you how to deal with crashes. I've installed <a href="https://github.com/me-no-dev/EspExceptionDecoder" target="_blank">Arduino ESP8266/ESP32 Exception Stack Trace Decoder</a> and ... nothing. I just could not find it in the menu. After searching through forums, I've found out it's because I use the new Arduino IDE v2 written in Node and the decoder is a Java plugin for Arduino IDE v1. There are some ways how to make it work with v2 IDE but for me the easiest way was to install IDE v1 (<a href="https://www.arduino.cc/en/software" rel="nofollow" target="_blank">1.8.19</a>). Finally, I had the <a href="https://github.com/CendaL/thermona-opentherm/blob/e0d1c9a371acf19800c96cb0a324972f5610b97f/crash/stack.trace.txt" target="_blank">stack trace</a>:</p>
<pre><code>
Exception 0: Illegal instruction
PC: 0x402010f4
EXCVADDR: 0x00000000
Decoding stack results
0x401001e2: OPENTHERM::_writeBit(unsigned char, unsigned char) at C:\Users\lucen\Downloads\Arduino\thermona.el9/opentherm.cpp line 204
0x4010041b: OPENTHERM::_timerISR() at C:\Users\lucen\Downloads\Arduino\thermona.el9/opentherm.cpp line 169
0x401004f4: timer1_isr_handler(void*, void*) at C:\Users\lucen\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.2\cores\esp8266\core_esp8266_timer.cpp line 37
0x4010053c: timer1_isr_handler(void*, void*) at C:\Users\lucen\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.2\cores\esp8266\core_esp8266_timer.cpp line 44
0x40101340: _stopPWM(uint8_t) at C:\Users\lucen\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.2\cores\esp8266\core_esp8266_waveform_pwm.cpp line 264
0x40101340: _stopPWM(uint8_t) at C:\Users\lucen\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.2\cores\esp8266\core_esp8266_waveform_pwm.cpp line 264
0x40100750: __digitalWrite(uint8_t, uint8_t) at C:\Users\lucen\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.2\cores\esp8266\core_esp8266_wiring_digital.cpp line 87
0x401001e2: OPENTHERM::_writeBit(unsigned char, unsigned char) at C:\Users\lucen\Downloads\Arduino\thermona.el9/opentherm.cpp line 204
0x401004bc: ets_post(uint8, ETSSignal, ETSParam) at C:\Users\lucen\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.2\cores\esp8266\core_esp8266_main.cpp line 181
0x401004f4: timer1_isr_handler(void*, void*) at C:\Users\lucen\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.2\cores\esp8266\core_esp8266_timer.cpp line 37
0x401004f4: timer1_isr_handler(void*, void*) at C:\Users\lucen\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.0.2\cores\esp8266\core_esp8266_timer.cpp line 37
</code></pre>
<p>But I was not clever from it... What illegal instruction during sending data to boiler? It's worth to note my knowledge about C++ is almost zero and the same applies for Arduino/WeMos HW. So I was not really sure what I need to look for...</p><p>I've enabled all compiler warnings in IDE Preferences and saw a warning about ICACHE_RAM_ATTR deprecation in favor of IRAM_ATTR. I have updated it in OpenTherm library but program still crashed. So I continued to search through forums. </p><p>Soon, I've found this <a href="https://stackoverflow.com/questions/58113937/esp8266-arduino-why-is-it-necessary-to-add-the-icache-ram-attr-macro-to-isrs-an" target="_blank">StackOverflow question</a> about ISR (Interrupt Service Routine) and why IRAM_ATTR attribute is needed. It did not ring the bell for me. Retrospectively, it should especially if I read it more carefully but I didn't and thus still not understand the issue. I just knew it has something to do with interrupts, functions not available in memory. Few hours forward and I found great blog post by Chris Dzombak: <a href="https://www.dzombak.com/blog/2021/10/Debugging-an-Intermittent-Arduino-ESP8266-ISR-Crash.html" target="_blank">Debugging an Intermittent Arduino/ESP8266 ISR Crash</a>. I had the same reset reason (<span style="font-family: courier;">rst cause:2, boot mode:(3,6)</span>) and Chris explained his investigation so well that even I have finally understand it.</p><p>I've went through all functions that were called from <span style="font-family: courier;">OPENTHERM::_timerISR()</span> method and add IRAM_ATTR attribute. I uploaded it to WeMos and ran it. It did successfully pass 2 hours timestamp. But it crashed again after about a day. Looking at the new crash I saw almost the same stack trace. I gave up and added IRAM_ATTR attribute to all private methods and public <span style="font-family: courier;">send</span> method (<a href="https://github.com/CendaL/arduino-opentherm/commit/d78c4dbafaf28ac53a1e8a3d2a560a52e62782b5" target="_blank">code</a>). Since then, the data has been flowing flawlessly for couple of days now, so it seems the problem has been fixed!</p>Lukáš Čenovskýhttp://www.blogger.com/profile/14627955690950344876noreply@blogger.com0tag:blogger.com,1999:blog-1384551653506604356.post-13185796852011063602023-01-02T00:09:00.013+01:002023-01-03T22:14:15.547+01:00Quest for lower electricity consumption - WeMos D1 Mini R2<a href="https://www.wemos.cc/en/latest/d1/d1_mini.html">WeMos </a><a href="https://www.wemos.cc/en/latest/d1/d1_mini.html">D1 Mini</a> is an ESP8266 WiFi board compatible with Arduino. <a href="https://github.com/jpraus/arduino-opentherm" target="_blank">OpenTherm</a> shield and library should support it so switch from Arduino Uno should be easy... Of course, it was not :-)<br /><br />First let's wire it correctly. I've seen <a href="https://github.com/jpraus/arduino-opentherm/issues/37">complain on github</a> it does not work with some pins. Here is my wiring which works for me:<br /><ul style="text-align: left;"><li>OpenTherm jumper set to 3.3V</li><li>OpenTherm D3 - WeMos D1 (via header) : <span style="font-family: courier;">#define BOILER_IN 5</span></li><li>OpenTherm D5 - WeMos D5 (via header) : <span style="font-family: courier;">#define BOILER_OUT 14</span></li><li>OpenTherm 3.3V - WeMos 3.3V</li><li>OpenTherm GND - WeMos GND</li></ul><div><br /></div><div>Picture which shows both leds blinking (ON and RxB leds) - proof it works:</div><div><br /></div><div><div class="separator" style="clear: both; text-align: left;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEiKUBibA1kT2w2wMtA9NBdni_dPXFmfyDbhxQg-mAKFZLf88zHwo7muzZ74LBBmHHbt_c7cnGcsYCLxmsDeP-rD2Bxr6jVOenVBZeZRhi1jVU44uO5EnMxtJWnY00aSpUTZuSQs-c1RiOsZBDFLtF6yP-mybC9pTnVW07yjg2RNNjbRw1b98aB_EbT4" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="363" data-original-width="582" height="400" src="https://blogger.googleusercontent.com/img/a/AVvXsEiKUBibA1kT2w2wMtA9NBdni_dPXFmfyDbhxQg-mAKFZLf88zHwo7muzZ74LBBmHHbt_c7cnGcsYCLxmsDeP-rD2Bxr6jVOenVBZeZRhi1jVU44uO5EnMxtJWnY00aSpUTZuSQs-c1RiOsZBDFLtF6yP-mybC9pTnVW07yjg2RNNjbRw1b98aB_EbT4=w640-h400" width="640" /></a></div><br />If you, for example, connect 5V from WeMos to 5V on OpenTherm shield, it will not work (it took me almost an hour to figure it out).</div><div><br /></div><div>My <a href="https://github.com/CendaL/thermona-opentherm/blob/83e073ac0a33fc5bf781d515cea52f83f5396b08/thermona-opentherm.ino" target="_blank">code</a> is based on examples how to connect to wifi, use https client and send a POST request (you can easily open them from Arduino IDE via menu File - Examples - <a href="https://github.com/esp8266/Arduino/tree/master/libraries/ESP8266HTTPClient/examples" target="_blank">ESP8266HTTPClient</a>).</div><div><br /></div><div>Data for every message type is sent to a webhook in Home Assistant (message id is suffix in webhook name). Sensor configurations are straightforward except conversion of bit mask to binary sensor:</div>
<pre><code>
template:
- trigger:
- platform: webhook
webhook_id: boiler0
sensor:
- name: Boiler - updated at
unique_id: boilerupdate
state: "{{ as_timestamp(now())|timestamp_custom('%d. %m. %Y %H:%M:%S') }}"
icon: mdi:clock-time-four-outline
binary_sensor:
- name: Boiler - error
unique_id: boiler0fault
state: "{{ trigger.json.data|int|bitwise_and(1) > 0 }}"
device_class: problem
- name: Boiler - central heating
unique_id: boiler0ch
state: "{{ trigger.json.data|int|bitwise_and(2) > 0 }}"
device_class: running
- name: Boiler - warm water
unique_id: boiler0dhw
state: "{{ trigger.json.data|int|bitwise_and(4) > 0 }}"
device_class: running
- name: Boiler - heating
unique_id: boiler0flame
state: "{{ trigger.json.data|int|bitwise_and(8) > 0 }}"
device_class: running
- trigger:
- platform: webhook
webhook_id: boiler18
sensor:
- name: Boiler - pressure
unique_id: boiler18
state: "{{ trigger.json.data }}"
state_class: measurement
device_class: pressure
unit_of_measurement: bar
icon: mdi:speedometer
...
</code></pre>
<div>Using all sensors, I've created a dashboard in Home Assistant to have a full picture how the boiler works :</div><div><br /></div><div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEhcOiEijllzw8pfq9RZQGxwCxQPwCZJ-SwzOsgNuaZVFkn6YY6WQ9bKPIyElGtjbyr92DmrIvGhCOGLO_9ckC0hjnpLaDpNLXVs39mVVuHXGRwkq0lFttUIuULPFQKnyZbjRDnXbIqWRZP45SbCyHyKgN8Xb-qYpnJyRLC29_2R_0RW5zrRPqXv4CTn" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="2160" data-original-width="2003" height="640" src="https://blogger.googleusercontent.com/img/a/AVvXsEhcOiEijllzw8pfq9RZQGxwCxQPwCZJ-SwzOsgNuaZVFkn6YY6WQ9bKPIyElGtjbyr92DmrIvGhCOGLO_9ckC0hjnpLaDpNLXVs39mVVuHXGRwkq0lFttUIuULPFQKnyZbjRDnXbIqWRZP45SbCyHyKgN8Xb-qYpnJyRLC29_2R_0RW5zrRPqXv4CTn=w595-h640" width="595" /></a></div><br />But my joy did not last long - within 1-2 hours, the data stopped flowing :-( Next post will be about figuring why?</div><div><br /></div><div>PS: The above image is from future when the issue is fixed.</div><div><br /></div>Lukáš Čenovskýhttp://www.blogger.com/profile/14627955690950344876noreply@blogger.com0tag:blogger.com,1999:blog-1384551653506604356.post-2433134074345169412022-12-30T21:00:00.006+01:002023-01-03T22:09:08.731+01:00Quest for lower electricity consumption - OpenTherm<p>The first article ends by the need to get more data from boiler because it heats warm water too often.</p><p>The first step I did was actually disabling circulating warm water every 15 minutes which I used to have always warm water immediately instead of waiting until it reaches a tap. Surprisingly, I observe almost no difference. There is still quite warm water immediately in the tap but the boiler heats water with the same frequency...</p><p>My home heating setup consists of two main heating branches - one with central heating (additionally split to radiators and floor heating) and the second with external tank for warm water. For some reason there is no three-way valve to stop flowing water to external tank when running in central heating mode. I suspect this is the culprit why boiler heats external tank approximately every two hours. To confirm it, I need data about what mode the boiler runs (although it can be guessed from the power consumption) and temperature in the external tank which triggers its heating. All can be retrieved via <a href="https://en.wikipedia.org/wiki/OpenTherm" rel="nofollow" target="_blank">OpenTherm</a> protocol.</p><p>OpenTherm standard is designed for communication between boiler and thermostat. Mine boiler - <a href="https://www.thermona.eu/en/electric-boilers/with-touch-display/boiler-therm-el-9" rel="nofollow" target="_blank">Thermona Therm EL 9</a> - does support it.</p><div>Googling how to get data from OpenTherm resulted in several links:</div><p></p><ol style="text-align: left;"><li><a href="https://www.home-assistant.io/integrations/opentherm_gw/">https://www.home-assistant.io/integrations/opentherm_gw/ </a><br />Hurray - there is a way how to get data to Home Assistant!</li><li><a href="http://otgw.tclcode.com/">http://otgw.tclcode.com/</a> - the gateway. Unfortunately, it looks quite complex...</li><li><a href="https://github.com/jpraus/arduino-opentherm">https://github.com/jpraus/arduino-opentherm</a> - library for Arduino by Jiří Praus with his HW looks much easier to use and I went for this option.</li><li>Only later I have found out it would be enough to use <a href="https://ihormelnyk.com/opentherm_adapter">https://ihormelnyk.com/opentherm_adapter</a> (because I don't need to communicate with thermostat).</li></ol><div>I have ordered <a href="https://www.tindie.com/products/jiripraus/opentherm-gateway-arduino-shield/" rel="nofollow" target="_blank">OpenTherm Gateway Arduino Shield</a>. I struggled to find 24V power supply, even wrote to Jiří who quickly suggested <a href="https://www.gme.cz/" rel="nofollow" target="_blank">GME</a> but eventually I've ordered it from <a href="https://tipa.eu/" rel="nofollow" target="_blank">Tipa</a>. Soldering it together was a challenge as I have soldered only big electronic parts till now. Because of my "huge" experiences, I've managed to solder headers upside down so I need to use wires to connect it to Arduino Uno for testing.</div><div><br /></div><div><div>Eventually, I was able to put everything together and <a href="https://github.com/jpraus/arduino-opentherm#testing-out-the-hardware" rel="nofollow" target="_blank">test</a> the hardware, interfaces and self-test. All tests have passed. The only issue/difference I saw was in step 6 where I measured ~4.7V instead of 5-7V.</div><div><br /></div></div><div>Here is the wiring because it was not obvious for me; pin constants in code are the same:</div><div><ul style="text-align: left;"><li>OpenTherm jumper set to 5V</li><li>OpenTherm D3 - Uno D3 (via header) : <span style="font-family: courier;">#define BOILER_IN 3</span></li><li>OpenTherm D5 - Uno D5 (via header) : <span style="font-family: courier;">#define BOILER_OUT 5</span></li><li>OpenTherm 5V - Uno 5V</li><li>OpenTherm GND - Uno GND</li></ul></div><div><div class="separator" style="clear: both; text-align: left;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEj8V7jRk7-V4wflwUtNENmSkie4dy0t8re2FDwdvoL6QS9OscxSioSDX-Nm4OIgYIfziz6WDgJ33XHrR2vN7N6ZI3OD1mUUjZe9kdevI9H693tkk4FEIyVG_EG2Gj1Y13Mg7Vxt38pZ2d2ZDXkzRwQkTI0WAOFqXaRfoiTcoKr6H-O8YQ6XNb8TF_TE" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="1037" data-original-width="1282" height="323" src="https://blogger.googleusercontent.com/img/a/AVvXsEj8V7jRk7-V4wflwUtNENmSkie4dy0t8re2FDwdvoL6QS9OscxSioSDX-Nm4OIgYIfziz6WDgJ33XHrR2vN7N6ZI3OD1mUUjZe9kdevI9H693tkk4FEIyVG_EG2Gj1Y13Mg7Vxt38pZ2d2ZDXkzRwQkTI0WAOFqXaRfoiTcoKr6H-O8YQ6XNb8TF_TE=w400-h323" width="400" /></a></div><br /></div><div>Having wired the OpenThem shield with Arduino, next step was to get data from boiler using <a href="https://github.com/CendaL/thermona-opentherm/blob/4ec2f8fdd2a6528e6a9e42c9fc8610f8c62e82bc/thermona-opentherm.ino" target="_blank">sample code</a>. The code uses <a href="https://github.com/jpraus/arduino-opentherm" target="_blank">arduino-opentherm</a> library which needs to be put into Arduino IDE Sketchbook library folder (e.g. <span style="font-family: courier;">C:\Users\{username}\Documents\Arduino\libraries\arduino-opentherm</span> on Windows).</div><div><br /></div><div><div class="separator" style="clear: both; text-align: left;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEi77DrWyf7pnatXLo7mVrcGv_Fwbg7eetcvdvJYnz_A25G73ssjGJ3kUve6xRsKButelRX3ewFEBTrLXi_3lp8p3If9VmjVrTiqJinUri7Nt4awpdsymNrQaeXTTIy7Zn7yoEDbH0nMEeYKenbDLp57_c13roIkcGSbG0glw4hu-QcVOZwYl4gAmfRV" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="386" data-original-width="775" height="159" src="https://blogger.googleusercontent.com/img/a/AVvXsEi77DrWyf7pnatXLo7mVrcGv_Fwbg7eetcvdvJYnz_A25G73ssjGJ3kUve6xRsKButelRX3ewFEBTrLXi_3lp8p3If9VmjVrTiqJinUri7Nt4awpdsymNrQaeXTTIy7Zn7yoEDbH0nMEeYKenbDLp57_c13roIkcGSbG0glw4hu-QcVOZwYl4gAmfRV" width="320" /></a></div><br />It worked! Going through <a href="https://github.com/jpraus/arduino-opentherm/blob/master/doc/Opentherm%20Protocol%20v2-2.pdf" target="_blank">OpenThem 2.2 specification</a>, I've identified useful messages for my boiler:</div><div><div><ul style="text-align: left;"><li>OT_MSGID_STATUS - boiler's status (what does it heat)</li><li>OT_MSGID_FAULT_FLAGS - current fault code</li><li>OT_MSGID_MODULATION_LEVEL - boiler's power (0-100%)</li><li>OT_MSGID_CH_WATER_PRESSURE - boiler's water pressure</li><li>OT_MSGID_FEED_TEMP - boiler's water temperature</li><li>OT_MSGID_DHW_TEMP - warm tank's water temperature</li><li>OT_MSGID_OUTSIDE_TEMP - outside temperature (for equithermal regulation)</li><li>OT_MSGID_DHW_SETPOINT - target water tank temperature</li></ul></div></div><div>Next step is to switch from Arduino Uno to WeMos D1 so I can send the data wirelessly to my Home Assistant.</div><p></p>Lukáš Čenovskýhttp://www.blogger.com/profile/14627955690950344876noreply@blogger.com0tag:blogger.com,1999:blog-1384551653506604356.post-19419551508525476422022-12-29T13:37:00.006+01:002023-01-03T22:00:56.341+01:00Quest for lower electricity consumption - MeasurementWith electricity prices skyrocketing this year, I've started looking into ways how to save some money by lowering electricity consumption. The first step is of course measurement!<div><br /></div><div>I use <a href="https://www.shelly.cloud/en-cz/products/product-overview/shelly-plus-2-pm" rel="nofollow" target="_blank">Shelly Plus 2PM</a> to control my external blinds so the choice was obvious - use <a href="https://www.shelly.cloud/en-cz/products/product-overview/shelly-3-em" rel="nofollow" target="_blank">Shelly 3EM</a>. To get some idea how installation looks like, I watched <a href="https://www.youtube.com/watch?v=CauzGDZUKwE" rel="nofollow" target="_blank">Shelly 3EM Installation video</a> by Jimmy James. Based on that, I've decided I can handle it myself.</div><div><br /></div><div style="text-align: left;"><b>Shelly 3EM Installation</b></div><div>The problem was space in our electrical cabinet. Because it's not only 1 spot for the meter but another 3 for the circuit breaker and place around input wires to put clamps. Eventually, I've hacked it for the beginning and put the circuit breaker outside of the cabinet. Just to see whether the meter works fine.</div><div><br /></div><div><div class="separator" style="clear: both; text-align: left;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEgmPyXaqIeT9-dK-yp66C6yd45sEBgF7vYvwuO1kpRqFcD5CIhmpHUnB8GKO6HEmHLkLrtZ1qJqjyRo3yf3Fla5WBYfzQXuQ182Ns-TEuLHwPyQq8Wczb2CJ-jPK9DeEvjtML_x6rQxRk5HM6ebscEkAkyN6v9l5O9ToHAuXn3P6Gee8NSpAJdzf8tM" style="margin-left: 1em; margin-right: 1em;"><img data-original-height="420" data-original-width="929" height="290" src="https://blogger.googleusercontent.com/img/a/AVvXsEgmPyXaqIeT9-dK-yp66C6yd45sEBgF7vYvwuO1kpRqFcD5CIhmpHUnB8GKO6HEmHLkLrtZ1qJqjyRo3yf3Fla5WBYfzQXuQ182Ns-TEuLHwPyQq8Wczb2CJ-jPK9DeEvjtML_x6rQxRk5HM6ebscEkAkyN6v9l5O9ToHAuXn3P6Gee8NSpAJdzf8tM=w640-h290" width="640" /></a></div><div><br /></div>It does! Of course, later I plan to make it nicer and hide the cables into the wall.</div><div><div style="text-align: left;"><br /></div><div style="text-align: left;"><b>Setup Shelly 3EM In Home Assistant</b></div><div>Setup in Shelly web interface is easy - I just restricted access for admin, configured wifi and added Device name - I called in Main Power so it will look good in Home Assistant. Then I added the Shelly integration to my Home assistant which configured automatically all sensors. I remember small hick-up with discovering the device - you need to enter just IP address and then it will ask you for user/password (I was trying to add http://IP which does not work).</div></div><div><br /></div><div><div class="separator" style="clear: both; text-align: left;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEjiKEzkLuGVSgBU6HFIf8ry_FTw4GWVhik987KT7qlAn647dcVdjpe8ZGzkDfpWXMjnbtazgdlfxRaiv0zZ5e79vIqVOuKopjm_ym_hSgXZegVhAmMw6EIuN7d6_RMgq3x4SRMNyTk5vZWdaSR60JsyjTaGSkkEKpmdoownwSFFRCcUCn_1rsMX597R" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="173" data-original-width="314" height="110" src="https://blogger.googleusercontent.com/img/a/AVvXsEjiKEzkLuGVSgBU6HFIf8ry_FTw4GWVhik987KT7qlAn647dcVdjpe8ZGzkDfpWXMjnbtazgdlfxRaiv0zZ5e79vIqVOuKopjm_ym_hSgXZegVhAmMw6EIuN7d6_RMgq3x4SRMNyTk5vZWdaSR60JsyjTaGSkkEKpmdoownwSFFRCcUCn_1rsMX597R=w200-h110" width="200" /></a></div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;">Later I have updated Shelly settings based on <a href="https://www.home-assistant.io/integrations/shelly/#shelly-device-configuration-generation-1" rel="nofollow" target="_blank">recommendation for generation 1 devices</a> and set <i>ColoT peer</i> directly to my Home assistant IP address. It seems to me it speeds up updates in Home Assistant.</div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;"><div class="separator" style="clear: both; text-align: left;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEib0aC-Ob6mG73Nivs8kQCfif3D-v-yjP0elcXb7n3CqJeNKJV3dowphYqePCelU74AOs5WjYEVKxcRXJ1JPrLeezX-hXMh4uhMykQrek7n0fZtx_EMm6GL6HKZo56C0VfleExfgYn2tDQHUeuD7g9bHpuO5C4O6wJD50nGmJXl2p09h9yOjtF381d1" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="157" data-original-width="194" height="162" src="https://blogger.googleusercontent.com/img/a/AVvXsEib0aC-Ob6mG73Nivs8kQCfif3D-v-yjP0elcXb7n3CqJeNKJV3dowphYqePCelU74AOs5WjYEVKxcRXJ1JPrLeezX-hXMh4uhMykQrek7n0fZtx_EMm6GL6HKZo56C0VfleExfgYn2tDQHUeuD7g9bHpuO5C4O6wJD50nGmJXl2p09h9yOjtF381d1=w200-h162" width="200" /></a></div><br /><div class="separator" style="clear: both;"><b>Energy Dashboard</b></div><div class="separator" style="clear: both;">For Energy dashboard in Home assistant, I just added all three channels/phases as <i>Grid consumption</i> - it nicely shows the consumption split by them.</div><div class="separator" style="clear: both;"><br /></div><div class="separator" style="clear: both;"><div class="separator" style="clear: both; text-align: left;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEgM2r_yB2Gh7GQ7RZp3bqHb5gAepmdjnmm3T3RgjnUqfBZiHhjv1Sr7b7uwFk-t_OZzU3wfqrYXDRVezqFAir3JvhF9ej7CQgzf9Wt-LojzJAZXqGMI0abzh3LzDlRb-ZfWcZp0kfDu6x0UlmxUN9clGHx75KO7sZwRtd5xTMVbfwYSgCUB8Y4yiRek" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="559" data-original-width="530" height="400" src="https://blogger.googleusercontent.com/img/a/AVvXsEgM2r_yB2Gh7GQ7RZp3bqHb5gAepmdjnmm3T3RgjnUqfBZiHhjv1Sr7b7uwFk-t_OZzU3wfqrYXDRVezqFAir3JvhF9ej7CQgzf9Wt-LojzJAZXqGMI0abzh3LzDlRb-ZfWcZp0kfDu6x0UlmxUN9clGHx75KO7sZwRtd5xTMVbfwYSgCUB8Y4yiRek=w380-h400" width="380" /></a></div><br /><div class="separator" style="clear: both;"><b>Detailed Energy Graphs</b></div><div class="separator" style="clear: both;">To see detailed consumption, first I set up <i>Total Power</i> sensor in configuration.yaml (based on Jimmy video):</div>
<pre><code>
template:
- name: Total Power
unique_id: total_power_tmpl
unit_of_measurement: W
state_class: measurement
device_class: energy
state: >
{% set phase1 = states('sensor.main_power_channel_a_power') | float %}
{% set phase2 = states('sensor.main_power_channel_b_power') | float %}
{% set phase3 = states('sensor.main_power_channel_c_power') | float %}
{{ phase1 + phase2 + phase3 }}
</code></pre>
<div class="separator" style="clear: both;">And then using <a href="https://github.com/RomRider/apexcharts-card" rel="nofollow" target="_blank">apexcharts-card graphs</a>, I've set up cards showing consumption in the last hour, 6 hours, etc.</div>
<pre><code>
type: custom:apexcharts-card
graph_span: 1h
header:
show: true
title: Energy 1h
all_series_config:
stroke_width: 1
series:
- entity: sensor.total_power
name: Total
color: rgb(162, 185, 207)
- entity: sensor.main_power_channel_a_power
name: Phase A
color: rgb(3, 169, 244)
- entity: sensor.main_power_channel_b_power
name: Phase B
color: rgb(128, 233, 234)
- entity: sensor.main_power_channel_c_power
name: Phase C
color: rgb(255, 191, 128)</code></pre>
<div class="separator" style="clear: both; text-align: left;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEjSlhdSwGRueZUCbfetInm3VlhL7JGiOznPq9lnpbUF8DU7R_KWRxFIdq6X00efCkREhE58fep-mUugfzQp2GFCLfzlDrEXol2vOldVVqYGO6BWidX-oRjSrBfICmA2w1UQ2f0IKoMqcGDw7ghWPbF0JPGZ48gex_qNTJ5UbmzbRl3vtzd-djAPTamI" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="353" data-original-width="973" height="232" src="https://blogger.googleusercontent.com/img/a/AVvXsEjSlhdSwGRueZUCbfetInm3VlhL7JGiOznPq9lnpbUF8DU7R_KWRxFIdq6X00efCkREhE58fep-mUugfzQp2GFCLfzlDrEXol2vOldVVqYGO6BWidX-oRjSrBfICmA2w1UQ2f0IKoMqcGDw7ghWPbF0JPGZ48gex_qNTJ5UbmzbRl3vtzd-djAPTamI=w640-h232" width="640" /></a></div><br /><div class="separator" style="clear: both;">The big spikes on 6h graph are when our boiler heats tank with warm water. The problem is it does it every ~2 hours even during night which is too often and thus costly…</div><div class="separator" style="clear: both;"><br /></div><div class="separator" style="clear: both;">The next part is to find out more data from the boiler.</div></div></div></div></div>Lukáš Čenovskýhttp://www.blogger.com/profile/14627955690950344876noreply@blogger.com0tag:blogger.com,1999:blog-1384551653506604356.post-80711071968578071402020-07-16T18:11:00.001+02:002020-07-16T18:13:02.899+02:00Unit Testing is OverratedI have found very good article about testing: <br />
<br />
<a href="https://dev.to/tyrrrz/unit-testing-is-overrated-150e" target="_blank">Unit Testing is Overrated</a><br />
<br />
<i>Here are the main takeaways:</i><br />
<i> </i><br />
<ol><li><i>Think critically and challenge best practices</i></li>
<li><i>Don't rely on the the test pyramid</i></li>
<li><i>Separate tests by functionality, rather than by classes, modules, or scope</i></li>
<li><i>Aim for the highest level of integration while maintaining reasonable speed and cost</i></li>
<li><i>Avoid sacrificing software design for testability</i></li>
<li><i>Consider mocking only as a last resort</i></li>
</ol><br />
I have two notes:<br />
<ol><li>Relying on acual GeoIP provider - I'd fake it for PR gates because we don't want to block our PRs when the provider has an outage. I keep using it for CD gates.</li>
<li>Caching test is not sufficient - I'd add assert the cache contains tested record.</li>
</ol><br />
The article links another good posts:<br />
<ul><li><a href="https://kentcdodds.com/blog/write-tests" target="_blank">Write tests. Not too many. Mostly integration.</a> - The Testing Trophy<br />
<i>It doesn't matter if your component renders component b with props c and d if component b actually breaks if prop e is not supplied. So while having some unit tests to verify these pieces work in isolation isn't a bad thing, it doesn't do you any good if you don't also verify that they work together properly. And you'll find that by testing that they work together properly, you often don't need to bother testing them in isolation.</i></li>
<li><a href="https://engineering.atspotify.com/2018/01/11/testing-of-microservices/" target="_blank">Testing of Microservices</a> - integration test is the king<i> </i></li>
</ul>Lukáš Čenovskýhttp://www.blogger.com/profile/14627955690950344876noreply@blogger.com0tag:blogger.com,1999:blog-1384551653506604356.post-37292701216752599572015-08-24T11:40:00.004+02:002015-08-24T11:41:15.641+02:00Lessons, Surprises, Mistakes - v2015<div lang="en-US" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-weight: bold;">Schedule</span></div>
<div lang="en-US" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Rewriting
the application always takes much longer time then you think. The POS was ready
for the new season but the admin part was missing and was only implemented
during the first month. On the other hand, the new architecture splits the
deliverable code to the kiosk and admin packages so admin updates do not affect
kiosks and the kiosk part is smaller.</div>
<div lang="en-US" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div lang="en-US" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-weight: bold;">Database structure</span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span lang="en-US">I
have the proper SQL initialization script (instead of creating the tables by
AMS). The database is normalized so getting data for reports is easier.
Previously, the transactions were stored in one table where items was just a
string of item ids. </span><span lang="cs">The </span><span lang="en-US">reason was
to have simple server code - I did not want to write server JavaScript code
which inserts data into two tables. Now, the Entity Framework handles all the
complexity so the transaction items are stored in separate table.</span></div>
<div lang="en-US" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div lang="en-US" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-weight: bold;">Query speed differences</span></div>
<div lang="en-US" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
I thought
there wouldn't be big difference when running SQL queries on my local SQL 2014
comparing to Azure - and if so, that Azure would be faster. But I was wrong.
For some reason, the same query via Entity Framework took couple of seconds on
local database but couple of minutes in Azure. I had no idea how to debug it
and find the reason so I rewrote the query into pure SQL which works fine.</div>
<div lang="en-US" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div lang="en-US" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-weight: bold;">Slow Android browser</span></div>
<div lang="en-US" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
When I
was testing the application on tablet, I saw it is very slow. After you touch a
button, you had to wait a little bit to be your next touch recognized.
Annoying. So I tried all available browsers for Android and the best one is
Opera. It hast the largest screen estate and it is really fast - see the
picture.</div>
<div lang="en-US" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgNSBXeotO0dP0dP7ZRwXkAxUFmDLHUlRtD40Qi0MurRhxQEn6sbky9xD9Mko9KelCZkIOvGkxjHOHcmA7akwJ_uOyiOrFsszdjyUHo03YbEkB7mOunWeNtiMbW_ke6Z_dIVTvmRr_dQlI/s1600/POSv2015.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="476" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgNSBXeotO0dP0dP7ZRwXkAxUFmDLHUlRtD40Qi0MurRhxQEn6sbky9xD9Mko9KelCZkIOvGkxjHOHcmA7akwJ_uOyiOrFsszdjyUHo03YbEkB7mOunWeNtiMbW_ke6Z_dIVTvmRr_dQlI/s640/POSv2015.png" width="640" /></a></div>
<div style="margin: 0in;">
<br /></div>
<div lang="en-US" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div lang="en-US" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-weight: bold;">Ghost transactions</span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span lang="en-US">When
my neighbor checked the sales one morning, he was very surprised. There were
sales for 450 000,- K</span><span lang="cs">č</span><span lang="en-US"> in one
kiosk. Quite a lot when a small ice cream costs 14,- K</span><span lang="cs">č</span><span lang="en-US">!</span><span lang="cs"> </span><span lang="en-US">It turned out that
the touch layer on the tablet went crazy and generated touch events on the
lower part of the display. The kiosk signed in a cashier at 3:15 by itself and
created one transaction every two seconds on average. Interesting bug - on the
other hand, it was nice load test :-).</span></div>
<div lang="en-US" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div lang="en-US" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-weight: bold;">Shift duplicate check</span></div>
<span style="font-family: Calibri; font-size: 11pt;">Kiosks
send all errors to the server and I check them from time to time. Early after
launch, there was an error that shift cannot be saved because it already exists
in database. It turned out I forgot duplicate error handling for new shifts. If
the shift was correctly stored in the database but the server response got
lost, kiosk tried to send the shift again and again.</span><br />
<div lang="en-US" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
Lukáš Čenovskýhttp://www.blogger.com/profile/14627955690950344876noreply@blogger.com0tag:blogger.com,1999:blog-1384551653506604356.post-4278796789059725182015-08-20T12:45:00.000+02:002015-08-20T12:45:32.647+02:00POS v2015<div lang="en-US" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
The new
version of POS should resolve several issues I had with the first version:</div>
<ul style="direction: ltr; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="disc">
<li lang="en-US" style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: Calibri; font-size: 11.0pt;">Smaller application size -
402 kB is quite a lot when my code is just 78kB</span></li>
<li lang="en-US" style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: Calibri; font-size: 11.0pt;">Admin part should work on
iPhone - as I wrote previously, this looks like Durandal issue</span></li>
<li lang="en-US" style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: Calibri; font-size: 11.0pt;">Resolve token expiration</span></li>
<li lang="en-US" style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: Calibri; font-size: 11.0pt;">Resolve the mysterious double
transactions</span></li>
</ul>
<div lang="en-US" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div lang="en-US" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Plus I
wanted to try new stuff:</div>
<ul style="direction: ltr; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="disc">
<li lang="en-US" style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: Calibri; font-size: 11.0pt;">Azure Mobile Services with
.NET backend</span></li>
<li lang="en-US" style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: Calibri; font-size: 11.0pt;">Gulp, Browserify</span></li>
<li lang="en-US" style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: Calibri; font-size: 11.0pt;">Mocha, Chai and Sinon for JS
testing</span></li>
</ul>
<div lang="en-US" style="font-family: Calibri; font-size: 11.0pt; margin-left: .375in; margin: 0in;">
<br /></div>
<div lang="en-US" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
The above
basically means to rewrite the app from scratch.</div>
<div lang="en-US" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div lang="en-US" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-weight: bold;">Smaller application size, iPhone</span></div>
<div lang="en-US" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
To
decrease application size, I decided to remove Durandal and jQuery and go with
just <a href="http://knockoutjs.com/">Knockout</a>. Communication among view
models/modules is done via <a href="http://www.knockmeout.net/2012/05/using-ko-native-pubsub.html">knockout-postbox</a>
- a simple pub/sub library. Other libraries I use are <a href="http://underscorejs.org/">Underscore.js</a> and <a href="http://momentjs.com/">Moment.js</a>. The final size is 220kB - almost a
half of the first version. The biggest part is of course AMS client library
with 134kB.</div>
<div lang="en-US" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div lang="en-US" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Removing
Durandal also removed sign-in issue on iPhone so my neighbor can finally use
his phone to check daily sales.</div>
<div lang="en-US" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div lang="en-US" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-weight: bold;">Resolve token expiration</span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span lang="en-US">In
the end, I've implemented the easier solution. Application remembers when the
login token was acquired and</span><span lang="cs"> </span><span lang="en-US">it
signs out kiosk user after 25 days as soon as there is no opened shift. Because
credentials are stored in the browser, it's just one more click for the
cashier.</span></div>
<div lang="en-US" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div lang="en-US" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-weight: bold;">Resolve the mysterious double transactions</span></div>
<div lang="en-US" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Of course
it was <a href="http://blog.codinghorror.com/the-first-rule-of-programming-its-always-your-fault/">my
fault</a>! When I performed deep analysis of duplicated transactions, I've
found out they are single-item transactions within sequence of single-item
transactions. And then it struck me - I
don't check whether save is in progress when saving new transaction. As the
transactions from local storage are sent in FIFO order, the same transaction
was sent several times to the server when cashier created multiple transactions
in row. Some of the duplications were caught by the server check for the
transaction duplication but some slipped through as saving data on server is
asynchronous.</div>
<div lang="en-US" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div lang="en-US" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
The new
version of course have the save in progress check and I have not seen any
duplicate transactions.</div>
<div lang="en-US" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div lang="en-US" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-weight: bold;">Azure Mobile Services with .NET backend</span></div>
<div lang="en-US" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Using
.NET backend in AMS server part is much easier for me then using JS backend.
One can use local server for development with local database so new
functionality development and testing is simpler. Also adding custom endpoints
is easy as it is almost normal Web Api application. I was hacking CRUD
endpoints in JS backend in past.</div>
<div lang="en-US" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div lang="en-US" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
The
backend uses Entity Framework to communicate with SQL server. I don't like EF
much. It's a black box to me - I don't know how it creates SQL queries and what
should I do to optimize them. So I rather use pure SQL queries for all custom
endpoints functionality (get data for reports).</div>
<div lang="en-US" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div lang="en-US" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-weight: bold;">Gulp, Browserify, Mocha, Chai, Sinon</span></div>
<div lang="en-US" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
I find <a href="http://gulpjs.com/">Gulp</a> easier to work with than working with <a href="http://gruntjs.com/">Grunt</a>. It's probably because I prefer to write
what I want to do then to define a configuration. Configuration is fine for
simple cases but for anything else, the list of commands to do is better.</div>
<div lang="en-US" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<br />
<div lang="en-US" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
I use <a href="http://browserify.org/">Browserify</a> to manage module dependencies and
create the final package. I found example how to use Browserify with <a href="https://mochajs.org/">Mocha</a> so I switched to Mocha from <a href="http://jasmine.github.io/">Jasmine</a>. It really does not matter which
test library one use as long as you test your code.</div>
<div lang="en-US" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
Lukáš Čenovskýhttp://www.blogger.com/profile/14627955690950344876noreply@blogger.com0tag:blogger.com,1999:blog-1384551653506604356.post-75831814719640618872014-08-12T22:00:00.000+02:002014-08-12T22:06:12.070+02:00Open issues<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">This is the last article for the moment. It is about things I don't know how to solve or have not yet had time to solve.</div><div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;"><br />
</div><div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;"><span style="font-weight: bold;">Double transactions</span></div><div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">I've already mentioned this <a href="http://gui-at.blogspot.cz/2014/08/lessons-surprises-mistakes-i-internet.html">issue</a> but I still don't know what is the root cause and how to mitigate it. The check to not insert already inserted transaction prevents creation of ~25 double transactions per day. Still, I see there is usually one double transaction per day (that is per ~2500 transactions). I am quite confident this is a sever side issue as <span style="font-family: "Courier New";">__createdAt</span> times are very close to each other.</div><div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;"><br />
</div><div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;"><span style="font-weight: bold;">POS does not work on iPhone</span></div><div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">For some reason, the application does not work on iPhone. The main page is displayed, user is correctly redirected to the Google's sign in page but after successful sing in, she is redirected back to the main page which does not proceed to the next page. I've briefly tested the issue on borrowed iPhone and found out it is caused by combination of AWS and Durandal. Pure AWS page works just fine and I have not found any indicia that Durandal does not work on iPhone. I believe I just need to borrow the iPhone for longer time to fix this issue.</div><div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;"><br />
</div><div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;"><span style="font-weight: bold;">Built-in *.azurewebsites.net SSL certificate</span></div><div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">The server is hosted on azurewebsites.net domain and uses built-in wildcard certificate for secure access. Although I have read several articles that using wildcard certificates is bad, I don't think it is bad in our case. It is a REST service where both endpoints are fixed. If you disagree, please let me know an example of real threat.</div><div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;"><br />
</div><div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">For reference:</div><ul style="direction: ltr; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="disc"><li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><a href="http://stackoverflow.com/questions/19357416/windows-azure-websites-https"><span style="font-family: Calibri; font-size: 11.0pt;">http://stackoverflow.com/questions/19357416/windows-azure-websites-https</span></a></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><a href="http://security.stackexchange.com/questions/8210/what-vulnerabilities-could-be-caused-by-a-wildcard-ssl-cert"><span style="font-family: Calibri; font-size: 11.0pt;">http://security.stackexchange.com/questions/8210/what-vulnerabilities-could-be-caused-by-a-wildcard-ssl-cert</span></a></li>
</ul><div style="font-family: Calibri; font-size: 11.0pt; margin-left: .375in; margin: 0in;"><br />
</div><div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;"><span style="font-weight: bold;">Authentication token expiration</span></div><div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">The authentication token expires approximately every month. Which is the reason why I still have not developed automatic re-authentication when the token expires. It hasn't had enough priority.</div><div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;"><br />
</div><div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">There is a way how to do it generally for all requests by using <a href="http://msdn.microsoft.com/en-us/library/azure/jj554202.aspx">withFilter</a> (<a href="http://blogs.msdn.com/b/carlosfigueira/archive/2014/02/24/using-service-filters-with-the-mobile-services-javascript-sdk.aspx">usage example</a>). Or I will just sign out and in every couple of days. Whatever will be easier to implement.</div><div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;"><br />
</div><div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;"><span style="font-weight: bold;">No tests for server side JavaScript</span></div><div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;"><span style="font-size: 11pt;">I have tests for client side JavaScript and for SQL queries but I do not have any tests for sever side JavaScript. The main reason is there was usually just a few lines of simple code so I did not bother to create a testing set up. But this is slowly changing so creating server side JavaScript tests climbs up to the top of my to-do list.</span></div><div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;"><br />
</div>Lukáš Čenovskýhttp://www.blogger.com/profile/14627955690950344876noreply@blogger.com0tag:blogger.com,1999:blog-1384551653506604356.post-67770964425954672782014-08-09T16:46:00.000+02:002015-01-24T16:07:42.808+01:00Lessons, Surprises, Mistakes III. - AMS<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Next set of lessons, surprises and mistakes; this time about Azure Mobile Services (AMS).</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-weight: bold;">Download location</span></div>
<div style="font-size: 11.0pt; margin: 0in;">
<span style="font-family: Calibri;">Maybe I am blind but I cannot find a download link for the actual version of </span><span style="font-family: "Courier New";">MobileServices.Web.js</span><span style="font-family: Calibri;"> script. WinJs versions are easily available via </span><a href="http://www.nuget.org/packages/WindowsAzure.MobileServices/"><span style="font-family: Calibri;">nuget</span></a><span style="font-family: Calibri;">. Digging into tutorials I have found a location with all Web versions. Comparing it with nuget page, the latest available Web version is 1.2.2 (although WinJs is already at 1.2.3):</span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<a href="http://ajax.aspnetcdn.com/ajax/mobileservices/MobileServices.Web-1.2.2.min.js">http://ajax.aspnetcdn.com/ajax/mobileservices/MobileServices.Web-1.2.2.min.js</a><br />
<br />
<i>Update 24.1.2015:</i> There is a <a href="https://github.com/Azure/azure-mobile-services/blob/master/CHANGELOG.md">changelog</a> from which you can see that at this moment, the latest Javascript SDK is <a href="http://ajax.aspnetcdn.com/ajax/mobileservices/MobileServices.Web-1.2.5.js">1.2.5</a>.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-weight: bold;">Shared Scripts</span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
There is a possibility to manually upload a script to shared folder on server:</div>
<div style="font-family: "Courier New"; font-size: 11.0pt; margin: 0in;">
azure mobile script upload <my_ams> shared/helper.js</my_ams></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
You can then use it via require in other scripts:</div>
<div style="font-family: "Courier New"; font-size: 11.0pt; margin: 0in;">
var helper = require('../shared/helper');</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Unfortunately, the scripts there are periodically deleted. It turned out that manual upload is not supported and you can only use this feature if you enable source control. See more info on <a href="http://social.msdn.microsoft.com/Forums/windowsazure/en-US/3aacc015-16c2-4d35-b7d1-b02ab8974d53/azure-service-issues-cannot-find-module-when-referencing-shared-script?forum=azuremobile">forum</a>.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-weight: bold;">Stuck scheduled tasks</span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
I've managed to get the scheduled AMS task into state where it ignored all my updates. It ran the old script version no matter what I did. Even deleting the task and creating a new one with different name did not help. I asked on <a href="http://social.msdn.microsoft.com/Forums/windowsazure/en-US/e9e54d40-7ec5-4f96-ba0a-45a450e83ef9/mobile-services-scheduler-bug?forum=azuremobile">forum </a>and got <span style="font-size: 11pt;">quickly</span><span style="font-size: 11pt;"> </span><span style="font-size: 11pt;">an advice how to work around the issue by restarting the whole mobile services.</span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-weight: bold;">Wrong authorization</span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Although I require an authorized user when accessing all AMS endpoints, it took me a while to spot an error in my implementation. I did not refuse cashier operations from unknown users. You couldn't do it from UI, of course. But any script kiddie could download the POS, get the application id, authorize with any google account, and send as many transactions/shifts as wanted. The fix was simple - only allow server operations to known users.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-weight: bold;">SQL Tables Initialization</span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
I let AMS to create my tables. It was a mistake. Only when I started to have performance issues with getting report data, I realized what types are used for the data. For example, as JavaScript does not have integers but only floats, all integer values were stored in float columns. Not very fast and index-able… I have changed type of some columns but next time I will create proper SQL init script.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-weight: bold;">SQL query optimizations</span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
I had not written complex SQL query for a long time so the first version of the report query was quite embarrassing. There were too many inner queries and other issues. After a consultation with my friend, I speeded up the query by</div>
<ul style="direction: ltr; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="disc">
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: Calibri; font-size: 11.0pt;">Use null id trick to </span><a href="http://stackoverflow.com/questions/7745609/sql-select-only-rows-with-max-value-on-a-column"><span style="font-family: Calibri; font-size: 11.0pt;">select the whole row with max value</span></a></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: Calibri; font-size: 11.0pt;">Use </span><span style="font-family: "Courier New"; font-size: 11.0pt;">union all</span><span style="font-family: Calibri; font-size: 11.0pt;"> instead of </span><span style="font-family: "Courier New"; font-size: 11.0pt;">union</span><span style="font-family: Calibri; font-size: 11.0pt;"> because all sub-queries are distinct</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: Calibri; font-size: 11.0pt;">Converting float columns to integer columns</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: Calibri; font-size: 11.0pt;">Index on </span><span style="font-family: "Courier New"; font-size: 11.0pt;">date</span><span style="font-family: Calibri; font-size: 11.0pt;"> column did not help because server still clustered index on </span><span style="font-family: "Courier New"; font-size: 11.0pt;">__createdAt</span><span style="font-family: Calibri; font-size: 11.0pt;"> column which has very similar content to the </span><span style="font-family: "Courier New"; font-size: 11.0pt;">date</span><span style="font-family: Calibri; font-size: 11.0pt;"> columns</span></li>
</ul>
Lukáš Čenovskýhttp://www.blogger.com/profile/14627955690950344876noreply@blogger.com0tag:blogger.com,1999:blog-1384551653506604356.post-27749124515416198632014-08-04T13:16:00.000+02:002014-08-04T13:16:07.875+02:00Lessons, Surprises, Mistakes II.<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Next set of lessons, surprises and mistakes; this time various issues in no particular order.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-weight: bold;">Hardware issues</span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
We use some cheap Android Sencor Element tablets. From 10 tablets, 2 were faulty and we had to replace them. You can recognize faulty tablet by its freezing. Yet there were also more interesting manifestations (appeared only on one tablet):</div>
<ul style="direction: ltr; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="disc">
<li><span style="font-family: Calibri; font-size: 11pt;">Time speeding up by minutes per hour. That produced transactions and shifts with wrong time. Automatic time synchronization does not help as I assume it is synchronized only once per day.</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: Calibri; font-size: 11.0pt;">Touch events were not registered in some areas of the screen or were shifted (touching on one place triggers action from different place).</span></li>
</ul>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-weight: bold;">User feedback</span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
It's always good to observe and talk to your very end users. Based on seeing the POS in action and talking to one cashier, I realized the buttons were too small. The cashier had hard time to aim at the right button. So I've made all buttons and texts as big as possible:<br />
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgdjfQPnyKPO5pdF0GTemJjfdndDDMTeeA9zSfQT1RArU1UruhpB5s7aoTb00Baqd43nsaEfBAJEca8mL8j4za_mlsMkFKBv4RRxr3DwpDNn1eoNbecnnK8apUNvo24wOslhYCvljIwLHE/s1600/pos.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgdjfQPnyKPO5pdF0GTemJjfdndDDMTeeA9zSfQT1RArU1UruhpB5s7aoTb00Baqd43nsaEfBAJEca8mL8j4za_mlsMkFKBv4RRxr3DwpDNn1eoNbecnnK8apUNvo24wOslhYCvljIwLHE/s1600/pos.png" height="450" width="640" /></a></div>
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-weight: bold;">Wrong development browser size</span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
I use Chrome <a href="https://chrome.google.com/webstore/detail/resolution-test/idhfcdbheobinplaamokffboaccidbal">Resolution Test plugin</a> to easily resize browser window to tablet's resolution (1024 x 786). But I made a stupid mistake. The POS web page on tablet spreads across the full tablet screen (minus Android menu bar) while the plugin resizes the whole browser window to the specified size. So page areas were different. To have the same area in browser and on tablet (1024 x 720), the correct resolution for the plugin is 1034 x 826.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-weight: bold;">Browser zooming</span></div>
<div style="font-size: 11pt; margin: 0in;">
<div>
<div style="font-family: Calibri;">
Cashiers had also problem with browser zooming triggered by double tap. For a long time, I did not know how to disable it because I thought it is as a browser functionality and looked for an option in browser. Eventually, I've found out it is a HTML5 meta viewport property and you can turn it off by:</div>
<span style="font-family: Courier New, Courier, monospace;"><meta
name="viewport" content="user-scalable=0" /></span></div>
<span style="font-family: Courier New, Courier, monospace;"></span></div>
<div style="font-family: "Courier New"; font-size: 11.0pt; margin: 0in;">
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-weight: bold;">JavaScript caching</span></div>
<div style="font-size: 11.0pt; margin: 0in;">
<span style="font-family: Calibri;">I was surprised that browser loads JavaScript files from cache even when you ask to bypass it (e.g. by Ctrl+F5). I have not found a way how to force reloading JavaScript files. As a fix, I append a timestamp to </span><span style="font-family: "Courier New";">main.js</span><span style="font-family: Calibri;"> in the release package (and </span><span style="font-family: "Courier New";">index.css</span><span style="font-family: Calibri;"> because it can be cached too).</span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-weight: bold;">Automatic updates</span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Uptake of a new version was slow. It was normal that an old version was used several days after the new version was released because page refresh was needed on POS and only company employees could do it. Luckily, there was no critical update needed.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
I've implemented automatic updates like this: successful ping request returns the actual server version. When it comes back to a tablet, it means there is internet connection (necessary because of mobile internet - <a href="http://gui-at.blogspot.com/2014/08/lessons-surprises-mistakes-i-internet.html">see the previous post</a>). When the server version does not match with the client version and there is no cashier signed in, the web page is refreshed which performs the update. The server version is set manually as part of release deployment.<br />
<br /></div>
Lukáš Čenovskýhttp://www.blogger.com/profile/14627955690950344876noreply@blogger.com0tag:blogger.com,1999:blog-1384551653506604356.post-35596191055303688272014-08-01T16:44:00.000+02:002014-08-01T16:45:33.750+02:00Lessons, Surprises, Mistakes I. - Internet Connection<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span lang="en-US">This is the first of several posts about my mistakes, surprises and lessons learned. I'm always eager to see other's experiences so I can learn from them. Here are mine</span><span lang="cs">s</span><span lang="en-US"> so you can learn from me.</span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-weight: bold;">Going Live</span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
1st April morning. It kind of works. Most kiosks are online and I see signed in cashiers and some transactions. Some kiosks are offline. I guessed internet connection problems.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span lang="en-US">It was very </span><span lang="cs">similar</span><span lang="en-US"> during the next couple of days</span><span lang="cs">. There were rare moments when all kiosk</span><span lang="en-US">s</span><span lang="cs"> were online. </span><span lang="en-US">M</span><span lang="cs">ost of the time </span><span lang="en-US">at least one </span><span lang="cs">kiosk</span><span lang="en-US"> was </span><span lang="cs">offline.</span><span lang="en-US"> We discussed the cause of the issues and i</span><span lang="cs">t was </span><span lang="en-US">clear that the culprit was the internet connection. The mobile internet is ***. It disconnects every now and then, it drops packets, it is sloooooow. Especially in the kiosk which is a big metal box. I had to mitigate the problems.</span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-weight: bold;">Status emails</span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
First step was to have better visibility of the problems. Having status page with kiosks overview was nice but it was not enough - nobody periodically checked it. That's why I've implemented status emails. AMS offers scheduled jobs and I use them to periodically check kiosk statuses and send warning email with problematic kiosks. Initially, the check was every 30 minutes but there were too many emails and nobody cared about them. Currently, the status is checked once per hour during the opening hours and a warning is sent only when kiosk does not ping for more than one hour.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-weight: bold;">Automatic router restarts</span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
We've discovered the biggest issue in routers. They disconnect from the internet quite often. Luckily for us, routers have monitoring functionality. They can ping specified servers and if no reply comes back, they restart themselves. Once turned on (with 30 minutes threshold), the internet reliability increased dramatically.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-weight: bold;">Inability to sign in/out</span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Pages in Durandal are stored in separate files. That means there has to be internet connection to successfully navigate to another page (e.g. starting/ending shift). I thought (but not tested) that browser would cache all pages. But I was wrong and with such bad internet connection, some cashiers could not sign in or out - the screen just turned white because the target page could not be loaded.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-size: 11.0pt; margin: 0in;">
<span style="font-family: Calibri;">The solution is to build the whole app into three files (</span><span style="font-family: "Courier New";">index.html</span><span style="font-family: Calibri;">, </span><span style="font-family: "Courier New";">index.css</span><span style="font-family: Calibri;">, and </span><span style="font-family: "Courier New";">main.js</span><span style="font-family: Calibri;">) which are loaded at once during POS initialization. To do that, I use </span><span style="font-family: "Courier New";">grunt-durandal</span><span style="font-family: Calibri;"> and </span><span style="font-family: "Courier New";">grunt-uncss</span><span style="font-family: Calibri;"> tasks when creating release package. The only communication then is sending data to server which is backed up in local storage.</span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-weight: bold;">Double transactions</span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Another sign of bad internet connection is double transactions. POS sends data, data is written into database, but the confirmation is lost on a way back and POS sends data again. I have actually seen several triple transactions in the database. To solve this issue, I have added a check before inserting a new data into database.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-size: 11.0pt; margin: 0in;">
<span style="font-family: Calibri;">It did not fully mitigated the issue though. I'm still seeing some duplicate transactions. AMS provides </span><span style="font-family: "Courier New";">createdAt</span><span style="font-family: Calibri;"> column and according to this column these duplicate transactions are usually milliseconds apart or even at the same time. I have no idea how these duplicate transactions are created. It can be strange network behavior, bug in AMS…</span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-weight: bold;">Success transaction visibility</span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-size: 11pt;">This is a small tweak. POS displays whether the last server operation was successful or not. It's not for cashiers because they don't care. It's mainly for company employees when they check the kiosk on site to know that everything is OK.</span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-weight: bold;">Summary</span><br />
<span style="font-size: 11pt;">After
implementing above fixes, the reliability is quite good. It is rare to see
offline kiosk. There is still some work I'd like to do though. For example, I do not properly handle authentication token expiration so somebody has to
manually re-login kiosks every ~30 days otherwise they would be offline because
of missing authentication.</span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
Lukáš Čenovskýhttp://www.blogger.com/profile/14627955690950344876noreply@blogger.com0tag:blogger.com,1999:blog-1384551653506604356.post-90915748596465427692014-07-29T23:57:00.000+02:002014-07-29T23:57:21.156+02:00POS Architecture<div style="font-family: Calibri; font-size: 11pt; margin: 0in;"><a href="https://www.blogger.com/blogger.g?blogID=1384551653506604356" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"></a>The initial design proved itself to be working nicely so I've stuck to it:</div><ul style="direction: ltr; unicode-bidi: embed;" type="disc"><li><span style="font-family: Calibri; font-size: 11pt;">Android tablet in each kiosk connected to the Internet via 3G Wi-Fi router. Tablets with 3G modem are more expensive and every kiosk had already been equipped with 3G Wi-Fi router because of the previous system.</span></li>
<li style="vertical-align: middle;"><span style="font-family: Calibri; font-size: 11.0pt;">HTML SPA (Single Page Application) as client POS application running in the browser on the tablet</span></li>
<li style="vertical-align: middle;"><span style="font-family: Calibri; font-size: 11pt;">JavaScript Azure Mobile Services (AMS) with SQL database as server</span></li>
</ul><span style="font-family: Calibri; font-size: 11pt;">The software architecture is guided by using Durandal (MVVM design pattern) on the client and AMS on the server. I decided to use Durandal because I like the testability of MVVM applications and AMS because of its easy use.</span><br />
<br />
<b>POS as web page</b><br />
<span style="font-family: Calibri; font-size: 11pt;">The whole POS software runs as a HTML5 application in full screen browser. Cashiers cannot use other apps on the tablet because the special tablet holder hides main Android menu, all hardware buttons and prevents edge swiping which is the only way how to control the browser. Here is the picture of the real device:</span><br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;"><br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://www.blogger.com/blogger.g?blogID=1384551653506604356" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"></a><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiahco5DkIYRB-Eh7hxb9WybpT7211RgiUsnx0URvC8UtUZXDCFZilkHIXjTAfVvBxwjDVhMT4vb04S-LsBUrFj2oEULCiYw2covPEfj7daK75CxW7kn-s477xQFwKKB3GFASpXTz-7yeM/s1600/POS.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img alt="POS" border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiahco5DkIYRB-Eh7hxb9WybpT7211RgiUsnx0URvC8UtUZXDCFZilkHIXjTAfVvBxwjDVhMT4vb04S-LsBUrFj2oEULCiYw2covPEfj7daK75CxW7kn-s477xQFwKKB3GFASpXTz-7yeM/s1600/POS.jpg" height="496" title="POS" width="640" /></a></div><br />
</div><b>Authentication</b><br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">All AMS endpoints require authenticated user. AMS supports several authentication providers. I've chosen Google provider as all company employees have Google account. Special Google account has been created for every kiosk. The kiosk account is signed in in the POS browser all the time so no cashier needs to know its password.</div><br />
<b>Authorization</b><br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">There are only two groups of users:</div><ol style="direction: ltr; font-family: Calibri; font-size: 11pt; unicode-bidi: embed;" type="1"><li style="vertical-align: middle;" value="1"><span lang="en-US" style="font-size: 11pt; font-style: italic;">Cashiers</span><span lang="en-US" style="font-size: 11pt;"><br />
They can only insert data (transactions, shifts)</span><span lang="cs" style="font-size: 11pt;">.</span><span lang="en-US" style="font-size: 11pt;"> Every cashier has assinged a PIN which is used to start her shift.</span></li>
<li style="vertical-align: middle;"><span style="font-size: 11pt; font-style: italic;">Company employees</span><span style="font-size: 11pt;"><br />
Additionally to cashiers, they can also edit some data (cashier management) and read the data (reports). As there are only 3 employees at this moment, their Google ids are hardcoded in scripts.</span></li>
</ol><br />
<b>Offline mode</b><br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">Because kiosks are connected to the Internet via mobile network, I expected they can be offline from time to time. That’s why every operation is first saved to browser local storage and only when server confirms it is saved in the database, it is deleted from the local storage. When confirmation comes, then next unsaved operation is sent to the server until the queue is empty. Operations are sent one by one to not trigger multiple requests at the same time (imagine sending all transactions from one day at once).</div><br />
<b>Operations</b><br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;"><span lang="en-US">As the launch day approached, I started to wonder/worry how to monitor the health of the kiosks. There is no way to connect to them from the Internet and check the POS status. So I did it other way round. Every POS pings the server in specified interval (10 minutes) and reports its status (signed in cashier, locally stored operations, POS version). Company employees and I have access to the web page which displays statuses of all kiosks. The most important information is whether kiosk is alive (pinging) or dead. Repair is simple - </span><a href="https://www.youtube.com/watch?v=p85xwZ_OLX0"><span lang="en-US">restart tablet </span><span lang="cs">and</span><span lang="en-US">/or router</span></a><span lang="en-US">.</span></div><br />
<b>Release package</b><br />
<div style="font-size: 11.0pt; margin: 0in;"><span style="font-family: Calibri;">Grunt task for creating release package contains steps to uglify JavaScript and uncss/mincss (uglify for CSS rules). Initially, the Durandal pages were separate files even in the release package but I had to change it (more on that later). Now, the whole POS consists of three files: </span><span style="font-family: "Courier New";">index.html</span><span style="font-family: Calibri;">, </span><span style="font-family: "Courier New";">index.css</span><span style="font-family: Calibri;">, and </span><span style="font-family: "Courier New";">main.js</span><span style="font-family: Calibri;">.</span></div><br />
Lukáš Čenovskýhttp://www.blogger.com/profile/14627955690950344876noreply@blogger.com0tag:blogger.com,1999:blog-1384551653506604356.post-82956353843833603752014-07-27T16:49:00.000+02:002014-07-27T16:49:45.434+02:00Development Environment<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">I took this project as an opportunity to learn recent web development techniques. This is the list of buzz words I've used:</div><ul style="direction: ltr; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="disc"><li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><a href="http://durandaljs.com/"><span style="font-family: Calibri; font-size: 11.0pt;">Durandal</span></a><span style="font-family: Calibri; font-size: 11.0pt;"> (it uses </span><a href="http://jquery.com/"><span style="font-family: Calibri; font-size: 11.0pt;">jQuery</span></a><span style="font-family: Calibri; font-size: 11.0pt;">, </span><a href="http://knockoutjs.com/"><span style="font-family: Calibri; font-size: 11.0pt;">KnockoutJs</span></a><span style="font-family: Calibri; font-size: 11.0pt;"> and </span><a href="http://requirejs.org/"><span style="font-family: Calibri; font-size: 11.0pt;">RequireJs</span></a><span style="font-family: Calibri; font-size: 11.0pt;">) for wiring up the application</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><a href="http://twitter.github.com/bootstrap/"><span style="font-family: Calibri; font-size: 11.0pt;">Bootstrap</span></a><span style="font-family: Calibri; font-size: 11.0pt;"> for styling</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><a href="https://azure.microsoft.com/en-us/services/mobile-services/"><span style="font-family: Calibri; font-size: 11.0pt;">Azure Mobile Service</span></a><span style="font-family: Calibri; font-size: 11.0pt;"> with SQL database for backend</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><a href="http://gruntjs.com/"><span style="font-family: Calibri; font-size: 11.0pt;">Grunt</span></a><span style="font-family: Calibri; font-size: 11.0pt;"> for all the boring repetitive development tasks</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><a href="http://bower.io/"><span style="font-family: Calibri; font-size: 11.0pt;">Bower</span></a><span style="font-family: Calibri; font-size: 11.0pt;"> for managing web packages</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><a href="http://jasmine.github.io/"><span style="font-family: Calibri; font-size: 11.0pt;">Jasmine</span></a><span style="font-family: Calibri; font-size: 11.0pt;"> for testing the application and </span><a href="http://www.jshint.com/"><span style="font-family: Calibri; font-size: 11.0pt;">JSHint</span></a><span style="font-family: Calibri; font-size: 11.0pt;"> to check my JavaScript files</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><a href="http://tst.codeplex.com/"><span style="font-family: Calibri; font-size: 11.0pt;">T.S.T.</span></a><span style="font-family: Calibri; font-size: 11.0pt;"> for testing SQL queries</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><a href="http://www.sublimetext.com/"><span style="font-family: Calibri; font-size: 11.0pt;">Sublime</span></a><span style="font-family: Calibri; font-size: 11.0pt;"> as text editor - I wanted to know whether it can replace my Vim</span></li>
</ul><div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;"><br />
</div><div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">First, I used Bower to download Durandal and Bootstrap and prepared some basic Grunt configuration (run web server for development). Then, I dived into Durandal and learned how to structure the application. I tried to scaffold the app structure with existing Durandal's Grunt configuration but eventually, I created everything from scratch using the scaffolding structure as a tutorial. One learns more with this approach.</div><div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;"><br />
</div><div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">Having basic app structure (2 pages wired together) and grunt configuration, I created my first Jasmine test. I wanted to practice TDD for development but I found out that it does not work well for me. I did not have any clear vision how things would work and I did a lot of changes during the development. That's why I have just classic unit tests. Once I am satisfied with a feature, I create unit tests for it to make sure I won't break it in future.</div><div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;"><br />
</div><div style="font-size: 11.0pt; margin: 0in;"><span style="font-family: Calibri;">Although Durandal uses AMD, I worried about referencing source files from tests. The research, experimentation and creating Grunt configuration took me some time. But it works nicely in the end. Grunt's Jasmine task references two </span><span style="font-family: "Courier New";">require.config.js</span><span style="font-family: Calibri;"> files. The first one is the main application </span><span style="font-family: "Courier New";">require.config.js</span><span style="font-family: Calibri;"> file and the second one is Jasmine's one with </span><span style="font-family: "Courier New";">baseUrl</span><span style="font-family: Calibri;"> pointing to application's folder. All unit tests run when any of my JavaScript files change.</span></div><div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;"><br />
</div><div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">The last thing to figure out was how to test SQL queries. I set it up only later when I was developing the report and it was obvious that with manual testing I would not be able to cover all the corner cases properly. I use T.S.T. and I set it up like unit tests. I have special query file which contains just the SQL query. Using simple script, this query is inserted into production and testing JavaScript template files. If any of these files change, tests are triggered. Of course, there is a Grunt task for firing it up.</div><br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">Note at the end: web technologies change rapidly. Today, instead of Durandal, I might use just Knockout 3 based on Steve Sanderson NDC presentation: <a href="http://vimeo.com/97519516">Architecting large Single Page Applications with Knockout.js</a>.</div><div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;"><br />
</div>Lukáš Čenovskýhttp://www.blogger.com/profile/14627955690950344876noreply@blogger.com0tag:blogger.com,1999:blog-1384551653506604356.post-42775277718354663902014-07-26T19:27:00.000+02:002014-07-29T23:58:22.938+02:00POS for Ice-cream Kiosks<div style="font-family: Calibri; font-size: 11pt; margin: 0in;">I have a neighbor who sells ice cream. He owns several kiosks in various towns around the country. Two years ago we talked and he asked how hard would be to develop a simple POS (Point Of Sale) for him. He was very unsatisfied with his solution. He had a general cash register from which he had to download all transactions every day otherwise they would disappear. It had functionality he did not need but also lacked some functionality he needed. And it was expensive.</div><div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;"><br />
</div><div style="font-family: Calibri; font-size: 11pt; margin: 0in;">I worked for Radiant Systems so I know a thing or two about POS systems. This one is very simple. All it has to do is to send transactions to the server. The items and prices are fixed during the year which means there is no inventory management. The only changing variables are cashiers but their management is simple - we need to track just name, pin and kiosk where the cashier work. Reporting consists of one report with sold items and shift durations. Overall, quite simple functionality, right?</div><div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;"><br />
</div><div style="font-family: Calibri; font-size: 11pt; margin: 0in;">Then I started thinking about the server requirements which is a grey area for me. I don't know enough about it, especially about operations. I played with Node in Azure VM a little bit but I was not sure enough. The project went to sleep at the end of the year 2012…</div><div style="font-family: Calibri; font-size: 11pt; margin: 0in;"><br />
</div><div style="font-family: Calibri; font-size: 11pt; margin: 0in;">To be resumed a year later. My neighbor was more angry about the cash registers. And there was a new feature in Azure - the Mobile Services. I don't need to deal with the server details any more. Exactly what I wanted. So in the autumn 2013, we agreed on an attempt to implement it and I started to work on a proof of concept.</div><div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;"><br />
</div><div style="font-family: Calibri; font-size: 11pt; margin: 0in;">Following the <a href="http://en.wikipedia.org/wiki/KISS_principle" rel="nofollow">KISS principle</a>, the initial design was as follows:</div><ul style="direction: ltr; unicode-bidi: embed;" type="disc"><li style="vertical-align: middle;"><span style="font-family: Calibri; font-size: 11.0pt;">Android tablet in each kiosk connected to the Internet via 3G modem</span></li>
<li style="vertical-align: middle;"><span style="font-family: Calibri; font-size: 11.0pt;">HTML SPA (Single Page Application) as client POS application running in the browser on the tablet</span></li>
<li style="vertical-align: middle;"><span style="font-family: Calibri; font-size: 11.0pt;">JavaScript Azure Mobile Services with SQL database as server</span></li>
</ul><div style="font-family: Calibri; font-size: 11pt; margin: 0in;"><br />
</div><div style="font-family: Calibri; font-size: 11pt; margin: 0in;">I prototyped an initial solution during the rest of the year 2013. Three screens (sign in, cash box, report), all in one big awful HTML page. But that is what prototypes are for - to see whether it can work, then throw it away and do it properly. Here is the main screen from the prototype:</div><div style="font-family: Calibri; font-size: 11pt; margin: 0in;"><br />
</div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgcaHj8P3A2KGEUYKJtz7_qtNnQzJci4CvnkKIAmzb3FL6QKW1RCEti8ladCmrTaTgY2t3X7R4XkLixsf0A7ssfWBiuMj8-SRSB-fsup-x7ryulAyqxq5Oors23vce5ntjXs5DNwfubHy0/s1600/POSPrototype.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgcaHj8P3A2KGEUYKJtz7_qtNnQzJci4CvnkKIAmzb3FL6QKW1RCEti8ladCmrTaTgY2t3X7R4XkLixsf0A7ssfWBiuMj8-SRSB-fsup-x7ryulAyqxq5Oors23vce5ntjXs5DNwfubHy0/s1600/POSPrototype.png" height="450" width="640" /></a></div><div style="font-family: Calibri; font-size: 11pt; margin: 0in;"><br />
</div><div class="separator" style="clear: both; text-align: center;"></div><div style="margin: 0in;"><span style="font-family: Calibri; font-size: 11pt;">We tested it on a tablet and it looked good - a </span><span style="color: #6aa84f; font-family: Calibri; font-size: 11pt; font-weight: bold;">green light</span><span style="font-family: Calibri; font-size: 11pt;"> for my little, fun, real, side project. I had three months to properly develop it during my spare time. Yes, it was optimistic :-)</span></div><div style="margin: 0in;"><span style="font-family: Calibri; font-size: 11pt;"><br />
</span></div>Lukáš Čenovskýhttp://www.blogger.com/profile/14627955690950344876noreply@blogger.com1tag:blogger.com,1999:blog-1384551653506604356.post-34886634191625545772011-11-13T20:06:00.000+01:002011-11-14T00:56:58.917+01:00Sharing .xaml in WPF and Silverlight<p>Sharing code between WPF and Silverlight is not difficult. Good how-to is in the <a href="http://msdn.microsoft.com/en-us/library/ff921109(v=PandP.40).aspx">Prism guide</a>. However sharing more complex <span class="tt">.xaml</span> is not so easy. You have to handle different namespaces and properties.
<p><b>Handling different namespaces</b>
<p>Silverlight toolkit components are accessible via special namespace. So if you want to use a <span class="tt">WrapPanel</span> in Silverlight, you write something like this:
<pre class="prettyprint">
<UserControl x:Class="UnifiedXaml.TestControl"
xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
xmlns:toolkit="http://schemas.microsoft.com/winfx/2006/xaml/presentation/toolkit">
<toolkit:WrapPanel></toolkit:WrapPanel>
</UserControl>
</pre>
<p>WPF world is easier:
<pre class="prettyprint">
<UserControl x:Class="UnifiedXaml.TestControl"
xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml">
<WrapPanel></WrapPanel>
</UserControl>
</pre>
<p>It's clear those two <span class="tt">.xaml</span> files are so similar that they should be just one file. Here is one tiny line that makes it possible:
<pre class="prettyprint">
namespace UnifiedXaml
{
public class MyWrapPanel: WrapPanel { }
}
</pre>
<p>From now on, you can share the <span class="tt">.xaml</span> file:
<pre class="prettyprint">
<UserControl x:Class="UnifiedXaml.TestControl"
xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
xmlns:ux="clr-namespace:UnifiedXaml">
<ux:MyWrapPanel></ux:MyWrapPanel>
</UserControl>
</pre>
<p><b>Handling different properties</b>
<p>When you follow Prism guidelines, you have separate projects for WPF and Silverlight parts. That means the WPF/SL application resources dictionaries are also separate. So the easiest way to handle different properties is to use styles. Shared <span class="tt">.xaml</span> file specifies just the control and its style. All necessary properties for WPF are defined in the style stored in the WPF application resources dictionary and the same is done for Silverlight.Lukáš Čenovskýhttp://www.blogger.com/profile/14627955690950344876noreply@blogger.com0tag:blogger.com,1999:blog-1384551653506604356.post-65150192688812520772010-09-10T20:25:00.000+02:002010-09-10T20:25:52.289+02:00Anonymous function as default argument in methodsI was quite surprised by Python recently when I tried to use anonymous function that uses a class method (not classmethod) as a default argument for another class method and it worked. Example is worth thousands words so here it is:
<pre class="prettyprint">
class ListObject(object):
items = ((1, 'one'), (2, 'two'), (3, 'three'))
def get_num(self, item):
return item[0]
def get_text(self, item):
return item[1]
def get_list(self, fn=lambda s, i: ListObject.get_num(s, i)):
return [fn(self, j) for j in self.items]
lo = ListObject()
print 'numbers:', lo.get_list()
print 'texts:', lo.get_list(ListObject.get_text)
print 'texts:', lo.get_list(lambda s, i: i[1])
print 'items:', lo.get_list(lambda s, i: i)
</pre>
When you run this piece of code you get what you want:
<pre class="prettyprint">
numbers: [1, 2, 3]
texts: ['one', 'two', 'three']
texts: ['one', 'two', 'three']
items: [(1, 'one'), (2, 'two'), (3, 'three')]
</pre>
By default, <span class="tt">get_list</span> method uses <span class="tt">get_num</span> method and apply it to all items. But you can supply your own fucntion to apply it on items.<p>
Of course, this is very stupid example but it shows the principle. And this principle is quite handy for UI automation I am currently working on :-)Lukáš Čenovskýhttp://www.blogger.com/profile/14627955690950344876noreply@blogger.com0tag:blogger.com,1999:blog-1384551653506604356.post-13644855962067001572010-05-16T16:44:00.004+02:002010-05-16T18:50:56.580+02:00Distributing Silverlight application written in IronPython<p>When you have Silverlight application written in IronPython, it is a good
idea to split it to several files so browser can cache them separately. Later,
when you change something in your application, users will download only a small
part. During my attemts with IronPython and Silverligt, I have found
several catches. That's why I describe here my way how to distribute IronPython
Silverlight application.</p>
<p>I distribute my application as one <span class="tt">.html</span> file, one
<span class="tt">.xap</span> file, and several <span class="tt">.zip</span>
files. I use <span class="tt">.zip</span> because IIS already knows what to do
with .zip files. The files are:<br />
<ol>
<li><span class="tt">index.html</span></li>
<li><span class="tt">app.xap</span></li>
<li><span class="tt">IronPython.zip</span> - contains files from <span class="tt">IronPython-2.6.1\Silverlight\bin</span>:
<pre class="prettyprint">
IronPython.dll
IronPython.Modules.dll
</pre></li>
<li><span class="tt">Microsoft.Scripting.zip</span> - contains files from <span class="tt">IronPython-2.6.1\Silverlight\bin</span>:
<pre>
Microsoft.Dynamic.dll
Microsoft.Scripting.dll
Microsoft.Scripting.Core.dll
Microsoft.Scripting.ExtensionAttribute.dll
Microsoft.Scripting.Silverlight.dll
</pre></li>
<li><span class="tt">SLToolkit.zip</span> - contains files form Silverlight toolkit or SDK; in our case just</li>
<pre>
System.Windows.Controls.dll
</pre></li>
</ol>
<p>Let's create a small application, that uses <i>ChildWindow</i> control from
<a href="http://silverlight.codeplex.com/">Silverlight toolkit</a>:</p>
<pre class="prettyprint">
C:\IronPython-2.6.1\Silverlight\script\sl.bat python childwindow
</pre>
Change the <span class="tt">app.py</span> and <span
class="tt">app.xml</span>:<br />
<p><span class="tt">app.py</span></p>
<pre class="prettyprint">
from System.Windows import Application
from System.Windows.Controls import UserControl
class App:
def __init__(self):
self.root = Application.Current.LoadRootVisual(UserControl(), "app.xaml")
a = App()
</pre>
<p><span class="tt">app.xaml</span></p>
<pre class="prettyprint">
<UserControl x:Class="System.Windows.Controls.UserControl"
xmlns="http://schemas.microsoft.com/client/2007"
xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
xmlns:controls="clr-namespace:System.Windows.Controls;assembly=System.Windows.Controls">
<controls:ChildWindow >
<StackPanel>
<TextBlock Text="Text in ChildWindow"/>
<Button x:Name="btnNewWindow" Content="New window"/>
</StackPanel>
</controls:ChildWindow>
</UserControl>
</pre>
<p>We don't want to Chiron automatically add necesary <span
class="tt">.dll</span> files into <span class="tt">.xap</span> so we have to
add our own <span class="tt">AppManifest.xaml</span> and <span
class="tt">languages.config</span> into <span class="tt">childwindow\app</span>
folder:</p>
<p><span class="tt">AppManifest.xaml</span></p>
<pre class="prettyprint">
<Deployment xmlns="http://schemas.microsoft.com/client/2007/deployment"
xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
RuntimeVersion="2.0.31005.0"
EntryPointAssembly="Microsoft.Scripting.Silverlight"
EntryPointType="Microsoft.Scripting.Silverlight.DynamicApplication"
ExternalCallersFromCrossDomain="ScriptableOnly">
<Deployment.Parts>
</Deployment.Parts>
<Deployment.ExternalParts>
<ExtensionPart Source="Microsoft.Scripting.zip" />
<ExtensionPart Source="SLToolkit.zip" />
</Deployment.ExternalParts>
</Deployment>
</pre>
<p><span class="tt">languages.config</span></p>
<pre class="prettyprint">
<Languages>
<Language names="IronPython,Python,py"
languageContext="IronPython.Runtime.PythonContext"
extensions=".py"
assemblies="IronPython.dll;IronPython.Modules.dll"
external="IronPython.zip"/>
</Languages>
</pre>
<p>Now create all three <span class="tt">.zip</span> files and add them into
<span class="tt">childwindow</span> folder.</p>
<p>To test the application with Chiron, run</p>
<pre class="prettyprint">
C:\IronPython-2.6.1\Silverlight\bin\Chiron.exe /e: /d:childwindow
</pre>
<p>The <span class="tt">/e:</span>
switch is important - it tells Chiron to not put any assembly into generated
<span class="tt">.xap</span>
file. Check the application on <a
href="http://localhost:2060/index.html">http://localhost:2060/index.html</a>.</p>
<p>To generate <span class="tt">.xap</span> file for distribution, run:</p>
<pre class="prettyprint">
C:\IronPython-2.6.1\Silverlight\bin\Chiron.exe /e: /d:childwindow\app /z:app.zap
</pre>
<p>If you want to use anything from external assemblies in the code, you have to
add manually reference to those assemblies. For example, if you want to add a
button that creates a new <i>ChildWindow</i>, you have to change you code like
this:</p>
<p><span class="tt">app.py</span></p>
<pre class="prettyprint">
from System.Windows import Application
from System.Windows.Controls import UserControl
class App:
def __init__(self):
self.root = Application.Current.LoadRootVisual(UserControl(), "app.xaml")
self.root.btnNewWindow.Click += self.OnClick
def OnClick(self, sender, event):
import clr
clr.AddReference('System.Windows.Controls')
from System.Windows.Controls import ChildWindow
self.root.panel.Children.Add(ChildWindow(Content='new window'))
a = App()
</pre>
<p><span class="tt">app.xaml</span></p>
<pre class="prettyprint">
<UserControl x:Class="System.Windows.Controls.UserControl"
xmlns="http://schemas.microsoft.com/client/2007"
xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
xmlns:controls="clr-namespace:System.Windows.Controls;assembly=System.Windows.Controls">
<controls:ChildWindow >
<StackPanel x:Name="panel">
<TextBlock Text="Text in ChildWindow"/>
<Button x:Name="btnNewWindow" Content="New window"/>
</StackPanel>
</controls:ChildWindow>
</UserControl>
</pre>
<p>If you comment out the <span class="tt">clr.AddReference</span> line, <span
class="tt">ImportError</span> appears. See the explanation in Jimmy's <a
href="http://lists.ironpython.com/pipermail/users-ironpython.com/2010-May/012804.html">email</a>.</p>
<p>You can download the example <a
href="http://gui-at.cendaweb.cz/2010/05/childwindow.zip">here</a> but note the
<span class="tt">.zip</span> files do not contain and <span
class="tt">.dlls</span>.</p>Lukáš Čenovskýhttp://www.blogger.com/profile/14627955690950344876noreply@blogger.com0tag:blogger.com,1999:blog-1384551653506604356.post-23193609144134724662010-05-12T10:53:00.006+02:002010-05-16T18:55:14.045+02:00Silverlight validation with IronPython<p>Validation support in Silverlight is done via <a
href="http://timheuer.com/blog/archive/2008/06/04/silverlight-introduces-visual-state-manager-vsm.aspx">Visual
State Manager</a>. All invalid fields have red rectangle around themselves.
Unfortunately, this does not work out of the box in IronPython. We have to push
it a little bit.</p>
<p>To demonstrate how, I have created a small example. Create a Silverlight app
template and change <span class="tt">app.py</span> and <span class="tt">app.xaml</span>:</p>
<pre class="prettyprint">
C:\IronPython-2.6.1\Silverlight\script\sl.bat python validation
</pre>
<span class="tt">app.py</span>
<pre class="prettyprint">
import clrtype
import pyevent
from System.Windows import Application
from System.Windows.Controls import UserControl
from System.ComponentModel import INotifyPropertyChanged, PropertyChangedEventArgs
class ValidationClass(INotifyPropertyChanged):
__metaclass__ = clrtype.ClrClass
PropertyChanged = None
def __init__(self, win):
self.win = win
self._text = 'text'
self.PropertyChanged, self._propertyChangedCaller = pyevent.make_event()
def add_PropertyChanged(self, value):
self.PropertyChanged += value
def remove_PropertyChanged(self, value):
self.PropertyChanged -= value
def OnPropertyChanged(self, propertyName):
if self.PropertyChanged is not None:
self._propertyChangedCaller(self, PropertyChangedEventArgs(propertyName))
@property
@clrtype.accepts()
@clrtype.returns(str)
def text(self):
return self._text
@text.setter
@clrtype.accepts(str)
@clrtype.returns()
def text(self, value):
if not value.startswith('text'):
raise Exception('Value must start with text!')
self._text = value
self.OnPropertyChanged('text')
class App:
def __init__(self):
self.root = Application.Current.LoadRootVisual(UserControl(), "app.xaml")
self.root.DataContext = ValidationClass(self.root)
App()</pre>
<span class="tt">app.xaml</span>
<pre class="prettyprint">
<UserControl x:Class="System.Windows.Controls.UserControl"
xmlns="http://schemas.microsoft.com/client/2007"
xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml">
<StackPanel>
<TextBox x:Name="tbValidate1" Width="100" Height="25"
Text="{Binding text, Mode=TwoWay, ValidatesOnExceptions=True,
NotifyOnValidationError=True}" />
<TextBox Width="100" Height="25" />
<TextBlock Text="{Binding text}" HorizontalAlignment="Center" />
</StackPanel>
</UserControl>
</pre>
<p>When you run this application (<span
class="tt">C:\IronPython-2.6.1\Silverlight\script\server.bat
/d:validation</span>), you'll find out the validation does not work. There is
no red rectangle when you enter wrong value; e.g. <i>wrong</i>.</p>
<p>Note the second empty TextBox is there so you can move focus out of the
first one to update bound property.</p>
<p>For whatever reason, the invalid component is not switched into invalid
state. Could be IronPython bug, could be something else. Anyway to fix it, you
have to switch the control into invalid state manually. Add the <span
class="tt">BindingValidationError</span> event:</p>
<pre class="prettyprint">
from System.Windows import VisualStateManager
from System.Windows.Controls import ValidationErrorEventAction
...
class App:
def __init__(self):
self.root = Application.Current.LoadRootVisual(UserControl(), "app.xaml")
self.root.DataContext = ValidationClass(self.root)
self.root.BindingValidationError += self.OnBindingValidationError
def OnBindingValidationError(self, sender, event):
if event.Action == ValidationErrorEventAction.Added:
VisualStateManager.GoToState(event.OriginalSource, 'InvalidUnfocused', True)
else:
VisualStateManager.GoToState(event.OriginalSource, 'Valid', True</pre>
<p>Now when you enter wrong value into TextBox, you can see red rectangle
around the control. You also see, the bound variable has the old, correct value
<i>text</i>:</p>
<img src="http://gui-at.cendaweb.cz/2010/05/SL_validation.png"/>
<p>You can download the whole source <a
href="http://gui-at.cendaweb.cz/2010/05/SL_validation.zip">here</a>.</p>Lukáš Čenovskýhttp://www.blogger.com/profile/14627955690950344876noreply@blogger.com0tag:blogger.com,1999:blog-1384551653506604356.post-56174651601351244092010-03-16T11:28:00.002+01:002010-05-16T18:57:44.467+02:00Parsing XML with XDocument<p>I needed to parse a XML document recently in Silverlight. Unfortunately,
Silverlight does not have <i>System.Xml.XmlDocument</i> type so you need to use
<i>System.Xml.Linq.XDocument</i>.</p>
<p>The following example works in Silverlight and with small change also in WPF.</p>
<pre class="prettyprint">
# encoding: utf-8
import clr
clr.AddReferenceToFile('System.Xml.Linq.dll')
from System.Xml.Linq import XDocument, XNamespace
content = """<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:media="http://search.yahoo.com/mrss"
xmlns:atom="http://www.w3.org/2005/Atom">
<channel>
<item>
<title>This is the title</title>
<media:description type="html"><p></p></media:description>
<link>html/dsc00001.html</link>
<media:thumbnail url="preview/dsc00001.jpg"/>
<media:content url="web/dsc00001.jpg"/>
</item>
<item>
<title></title>
<media:description type="html"><p></p></media:description>
<link>html/dsc00002.html</link>
<media:thumbnail url="preview/dsc00002.jpg"/>
<media:content url="web/dsc00002.jpg"/>
</item>
</channel>
</rss>"""
xDoc = XDocument().Parse(content)
namespace = XNamespace.Get("http://search.yahoo.com/mrss")
for item in xDoc.Element('rss').Element('channel').Elements('item'):
print item.Element('title').Value
print item.Element(namespace+'thumbnail').Attribute('url').Value
</pre>
Here is the output:
<pre class="prettyprint">
This is the title
preview/dsc00001.jpg
preview/dsc00002.jpg
</pre>
<p>You have to have <span class="tt">System.Xml.Linq.dll</span> from
Silverlight SDK next to your <span class="tt">app.py</span>.</p>
<p>The change for WPF:</p>
<pre class="prettyprint">
clr.AddReference('System.Xml.Linq')
</pre>
<p>Also make sure you don't have Silverlight's <span
class="tt">System.Xml.Linq.dll</span> next to your script.</p>Lukáš Čenovskýhttp://www.blogger.com/profile/14627955690950344876noreply@blogger.com0tag:blogger.com,1999:blog-1384551653506604356.post-34506237235876113532010-02-23T17:21:00.003+01:002010-05-16T19:01:00.715+02:00Disk Encryption in OpenWrt<p>I have an external USB disk connected to my Asus WL-500gP which runs
OpenWrt KAMIKAZE (8.09.1, r16278).</p>
<p>I want to have the disk encrypted. So I installed every possible package
that has <i>aes</i> or <i>loop</i> in name but I still got the following error
message:</p>
<pre class="prettyprint">
root@OpenWrt:~# losetup -e aes /dev/loop/0 /dev/scsi/host0/bus0/target0/lun0/part1
Password:
ioctl: LOOP_SET_STATUS: Invalid argument
</pre>
<p>The solution was easy - install <i>losetup</i> that supports aes encryption.
You can find the package on <a
href="http://www.roth.lu/download/openwrt-usb-raid1-loopaes/">http://www.roth.lu/download/openwrt-usb-raid1-loopaes/</a>.
Download it, install it and viola - it works :-)</p>
<pre class="prettyprint">
root@OpenWrt:~# losetup -e aes /dev/loop/0 /dev/scsi/host0/bus0/target0/lun0/part1
Password:
root@OpenWrt:~# mkfs.ext2 -vm0 /dev/loop/0
mkdir /mnt/disk/
mount /dev/loop/0 /mnt/disc
</pre>
To mount the disc in script use
<pre class="prettyprint">
echo password | losetup -e aes /dev/loop/0 /dev/scsi/host0/bus0/target0/lun0/part1 -p0
</pre>Lukáš Čenovskýhttp://www.blogger.com/profile/14627955690950344876noreply@blogger.com1tag:blogger.com,1999:blog-1384551653506604356.post-81413261261521575732010-01-21T12:15:00.089+01:002010-05-16T19:06:27.146+02:00WinDbg for beginners<p>A few days ago I have encountered a strange problem. When I copied <span
class="tt">IronPython.Modules.dll</span> to my application's folder, it
freezed. Strange. It looked like a <a
href="http://ironpython.codeplex.com/WorkItem/View.aspx?WorkItemId=25933">bug</a>
in IronPython for me but it wasn't. With Dino Viehland help, I have found the
bug in Delphi code. Dino used <b>WinDbg</b> to discover what is going on in the
code. I show you it is not so hard to use it.</p>
<p>WinDbg is part of <a
href="http://www.microsoft.com/whdc/devtools/debugging/default.mspx">Debuging
Tolls for Windows</a>. Download and install appropriate version for your
Windows (32 or 64 bit). And download the <a
href="http://gui-at.cendaweb.cz/2010/01/WinDbgBug.zip">bugged
application</a>.</p>
<p>The <span class="tt">IpyTest.exe</span> is Delphi 7 application that creates
the CLR and runs function <span class="tt">Add3</span> from <span
class="tt">Host.dll</span> inside the CLR. The <span class="tt">Add3</span>
function creates IronPython engine and returns result of adding number 3 to the
input variabl. So when you run <span class="tt">IpyTest.exe</span> you should
see</p>
<pre class="prettyprint">
C:\IronPythonBug\Delphi>IpyTest.exe
CLRVersion = v2.0.50727
8 + 3 = 11
</pre>
<p>You may actually see it because this bug does not appear on all computers.
But on mine, I see just the CLRVersion. Then the program freezes.</p>
<p>Let's debug the application in WinDbg:</p>
<pre class="prettyprint">
"C:\Program Files\Debugging Tools for Windows (x86)\windbg.exe" C:\IronPythonBug\Delphi\IpyTest.exe
</pre>
<p>When WinDbg and <span class="tt">IpyTest.exe</span> start, you see this:</p>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://gui-at.cendaweb.cz/2010/01/WinDbgStart.png" imageanchor="1"
style="margin-left: 1em; margin-right: 1em;"><img border="0" height="310"
src="http://gui-at.cendaweb.cz/2010/01/WinDbgStart.png" width="400" /></a><br
/></div>
<p>Enter the following commands:</p>
<pre class="prettyprint">
0:000> sxe clr
0:000> sxe av
0:000> .symfix
0:000> .reload
Reloading current modules
............
</pre>
<p>If you want to know what these commands do, check the help :-) Then run the
debugger with <span class="tt">g</span> command. It runs for a while and then
it freezes with the following output:</p>
<pre class="prettyprint">
0:000> g
ModLoad: 76390000 763ad000 C:\WINDOWS\system32\IMM32.DLL
ModLoad: 79000000 79046000 C:\WINDOWS\system32\mscoree.dll
ModLoad: 77f60000 77fd6000 C:\WINDOWS\system32\SHLWAPI.dll
ModLoad: 79e70000 7a400000 c:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\mscorwks.dll
ModLoad: 78130000 781cb000 C:\WINDOWS\WinSxS\x86_Microsoft.VC80.CRT_1fc8b3b9a1e18e3b_8.0.50727.3053_x-ww_b80fa8ca\MSVCR80.dll
ModLoad: 7c9c0000 7d1d7000 C:\WINDOWS\system32\shell32.dll
ModLoad: 773d0000 774d3000 C:\WINDOWS\WinSxS\x86_Microsoft.Windows.Common-Controls_6595b64144ccf1df_6.0.2600.5512_x-ww_35d4ce83\comctl32.dll
ModLoad: 5d090000 5d12a000 C:\WINDOWS\system32\comctl32.dll
ModLoad: 790c0000 79bb7000 C:\WINDOWS\assembly\NativeImages_v2.0.50727_32\mscorlib\6d667f19d687361886990f3ca0f49816\mscorlib.ni.dll
ModLoad: 79060000 790bb000 c:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\mscorjit.dll
ModLoad: 33200000 3322c000 Microsoft.Scripting.dll
ModLoad: 68000000 68036000 C:\WINDOWS\system32\rsaenh.dll
ModLoad: 64020000 64033000 c:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\mscorsec.dll
ModLoad: 76c30000 76c5e000 C:\WINDOWS\system32\WINTRUST.dll
ModLoad: 77a80000 77b15000 C:\WINDOWS\system32\CRYPT32.dll
ModLoad: 77b20000 77b32000 C:\WINDOWS\system32\MSASN1.dll
ModLoad: 76c90000 76cb8000 C:\WINDOWS\system32\IMAGEHLP.dll
ModLoad: 74e30000 74e9d000 C:\WINDOWS\system32\RichEd20.dll
ModLoad: 04a30000 04cf5000 C:\WINDOWS\system32\xpsp2res.dll
ModLoad: 769c0000 76a74000 C:\WINDOWS\system32\userenv.dll
ModLoad: 5b860000 5b8b5000 C:\WINDOWS\system32\netapi32.dll
ModLoad: 75e60000 75e73000 C:\WINDOWS\system32\cryptnet.dll
ModLoad: 76bf0000 76bfb000 C:\WINDOWS\system32\PSAPI.DLL
ModLoad: 722b0000 722b5000 C:\WINDOWS\system32\SensApi.dll
ModLoad: 4d4f0000 4d549000 C:\WINDOWS\system32\WINHTTP.dll
ModLoad: 76f60000 76f8c000 C:\WINDOWS\system32\WLDAP32.dll
ModLoad: 33200000 3322c000 C:\IronPythonBug\Delphi\Microsoft.Scripting.dll
ModLoad: 34700000 34860000 IronPython.dll
ModLoad: 34700000 34860000 C:\IronPythonBug\Delphi\IronPython.dll
ModLoad: 33000000 33064000 Microsoft.Scripting.Core.dll
ModLoad: 33000000 33064000 C:\IronPythonBug\Delphi\Microsoft.Scripting.Core.dll
ModLoad: 7a440000 7abc5000 C:\WINDOWS\assembly\NativeImages_v2.0.50727_32\System\80978a322d7dd39f0a71be1251ae395a\System.ni.dll
ModLoad: 33400000 334dc000 Microsoft.Dynamic.dll
ModLoad: 33400000 334dc000 C:\IronPythonBug\Delphi\Microsoft.Dynamic.dll
ModLoad: 73000000 73008000 Microsoft.Scripting.ExtensionAttribute.dll
ModLoad: 73000000 73008000 C:\IronPythonBug\Delphi\Microsoft.Scripting.ExtensionAttribute.dll
ModLoad: 34c10000 34c80000 IronPython.Modules.dll
ModLoad: 34c10000 34c80000 C:\IronPythonBug\Delphi\IronPython.Modules.dll
ModLoad: 64890000 64981000 C:\WINDOWS\assembly\NativeImages_v2.0.50727_32\System.Configuration\b82c00e2d24305ad6cb08556e3779b75\System.Configuration.ni.dll
ModLoad: 637a0000 63cd6000 C:\WINDOWS\assembly\NativeImages_v2.0.50727_32\System.Xml\773a9786013451d3baaeff003dc4230f\System.Xml.ni.dll
(b6c.ef0): Unknown exception - code c0000091 (first chance)
>>> many lines omitted <<<
(b6c.ef0): Unknown exception - code c0000091 (first chance)
</pre>
<p>The <span class="tt">IpyTest.exe</span> window contains just a text:
<i>CLRVersion = v2.0.50727</i>. Break the debugger with <i>Ctrl-Break</i>. We
see there is an unknown exception with code <i>c0000091</i>. Let's restart the
program and tell WinDbg to stop on this exception with the following
commands:</p>
<pre class="prettyprint">
.restart
sxe c0000091
.symfix
.reload
g
</pre>
<p>Here is the output:</p>
<pre class="prettyprint">
0:003> .restart
CommandLine: C:\IronPythonBug\Delphi\IpyTest.exe
Symbol search path is: srv*
Executable search path is:
ModLoad: 00400000 00455000 image00400000
ModLoad: 7c900000 7c9af000 ntdll.dll
ModLoad: 7c800000 7c8f6000 C:\WINDOWS\system32\kernel32.dll
ModLoad: 7e410000 7e4a1000 C:\WINDOWS\system32\user32.dll
ModLoad: 77f10000 77f59000 C:\WINDOWS\system32\GDI32.dll
ModLoad: 77dd0000 77e6b000 C:\WINDOWS\system32\advapi32.dll
ModLoad: 77e70000 77f02000 C:\WINDOWS\system32\RPCRT4.dll
ModLoad: 77fe0000 77ff1000 C:\WINDOWS\system32\Secur32.dll
ModLoad: 77120000 771ab000 C:\WINDOWS\system32\oleaut32.dll
ModLoad: 77c10000 77c68000 C:\WINDOWS\system32\msvcrt.dll
ModLoad: 774e0000 7761d000 C:\WINDOWS\system32\ole32.dll
ModLoad: 77c00000 77c08000 C:\WINDOWS\system32\version.dll
(ea8.a14): Break instruction exception - code 80000003 (first chance)
eax=00351ec4 ebx=7ffd4000 ecx=00000004 edx=00000010 esi=00351f98 edi=00351ec4
eip=7c90120e esp=0012fb20 ebp=0012fc94 iopl=0 nv up ei pl nz na po nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000202
ntdll!DbgBreakPoint:
7c90120e cc int 3
0:000> sxe c0000091
0:000> .symfix
0:000> .reload
Reloading current modules
............
0:000> g
ModLoad: 76390000 763ad000 C:\WINDOWS\system32\IMM32.DLL
ModLoad: 79000000 79046000 C:\WINDOWS\system32\mscoree.dll
ModLoad: 77f60000 77fd6000 C:\WINDOWS\system32\SHLWAPI.dll
ModLoad: 79e70000 7a400000 c:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\mscorwks.dll
ModLoad: 78130000 781cb000 C:\WINDOWS\WinSxS\x86_Microsoft.VC80.CRT_1fc8b3b9a1e18e3b_8.0.50727.3053_x-ww_b80fa8ca\MSVCR80.dll
ModLoad: 7c9c0000 7d1d7000 C:\WINDOWS\system32\shell32.dll
ModLoad: 773d0000 774d3000 C:\WINDOWS\WinSxS\x86_Microsoft.Windows.Common-Controls_6595b64144ccf1df_6.0.2600.5512_x-ww_35d4ce83\comctl32.dll
ModLoad: 5d090000 5d12a000 C:\WINDOWS\system32\comctl32.dll
ModLoad: 790c0000 79bb7000 C:\WINDOWS\assembly\NativeImages_v2.0.50727_32\mscorlib\6d667f19d687361886990f3ca0f49816\mscorlib.ni.dll
ModLoad: 79060000 790bb000 c:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\mscorjit.dll
ModLoad: 33200000 3322c000 Microsoft.Scripting.dll
ModLoad: 68000000 68036000 C:\WINDOWS\system32\rsaenh.dll
ModLoad: 64020000 64033000 c:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\mscorsec.dll
ModLoad: 76c30000 76c5e000 C:\WINDOWS\system32\WINTRUST.dll
ModLoad: 77a80000 77b15000 C:\WINDOWS\system32\CRYPT32.dll
ModLoad: 77b20000 77b32000 C:\WINDOWS\system32\MSASN1.dll
ModLoad: 76c90000 76cb8000 C:\WINDOWS\system32\IMAGEHLP.dll
ModLoad: 74e30000 74e9d000 C:\WINDOWS\system32\RichEd20.dll
ModLoad: 04a30000 04cf5000 C:\WINDOWS\system32\xpsp2res.dll
ModLoad: 769c0000 76a74000 C:\WINDOWS\system32\userenv.dll
ModLoad: 5b860000 5b8b5000 C:\WINDOWS\system32\netapi32.dll
ModLoad: 75e60000 75e73000 C:\WINDOWS\system32\cryptnet.dll
ModLoad: 76bf0000 76bfb000 C:\WINDOWS\system32\PSAPI.DLL
ModLoad: 722b0000 722b5000 C:\WINDOWS\system32\SensApi.dll
ModLoad: 4d4f0000 4d549000 C:\WINDOWS\system32\WINHTTP.dll
ModLoad: 76f60000 76f8c000 C:\WINDOWS\system32\WLDAP32.dll
ModLoad: 33200000 3322c000 C:\IronPythonBug\Delphi\Microsoft.Scripting.dll
ModLoad: 34700000 34860000 IronPython.dll
ModLoad: 34700000 34860000 C:\IronPythonBug\Delphi\IronPython.dll
ModLoad: 33000000 33064000 Microsoft.Scripting.Core.dll
ModLoad: 33000000 33064000 C:\IronPythonBug\Delphi\Microsoft.Scripting.Core.dll
ModLoad: 7a440000 7abc5000 C:\WINDOWS\assembly\NativeImages_v2.0.50727_32\System\80978a322d7dd39f0a71be1251ae395a\System.ni.dll
ModLoad: 33400000 334dc000 Microsoft.Dynamic.dll
ModLoad: 33400000 334dc000 C:\IronPythonBug\Delphi\Microsoft.Dynamic.dll
ModLoad: 73000000 73008000 Microsoft.Scripting.ExtensionAttribute.dll
ModLoad: 73000000 73008000 C:\IronPythonBug\Delphi\Microsoft.Scripting.ExtensionAttribute.dll
ModLoad: 34c10000 34c80000 IronPython.Modules.dll
ModLoad: 34c10000 34c80000 C:\IronPythonBug\Delphi\IronPython.Modules.dll
ModLoad: 64890000 64981000 C:\WINDOWS\assembly\NativeImages_v2.0.50727_32\System.Configuration\b82c00e2d24305ad6cb08556e3779b75\System.Configuration.ni.dll
ModLoad: 637a0000 63cd6000 C:\WINDOWS\assembly\NativeImages_v2.0.50727_32\System.Xml\773a9786013451d3baaeff003dc4230f\System.Xml.ni.dll
(ea8.a14): Unknown exception - code c0000091 (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
eax=ffffff00 ebx=02643d64 ecx=02640010 edx=02648e3c esi=02643bd4 edi=02640010
eip=79098f48 esp=0012ce30 ebp=0012ce34 iopl=0 nv up ei pl nz na po nc
cs=001b ss=0023 ds=0023 es=0023 fs=0038 gs=0000 efl=00010202
mscorjit!Compiler::FlatFPIsSameAsFloat+0xa:
79098f48 d945fc fld dword ptr [ebp-4] ss:0023:0012ce30=02640010</pre>
<p>WinDbg stopped on the exception. Nice. And what now? <b>Analyze</b> the exception!</p>
<pre class="prettyprint">
0:000> !analyze -v
*******************************************************************************
* *
* Exception Analysis *
* *
*******************************************************************************
*** WARNING: Unable to verify checksum for C:\WINDOWS\assembly\NativeImages_v2.0.50727_32\mscorlib\6d667f19d687361886990f3ca0f49816\mscorlib.ni.dll
*** WARNING: Unable to verify checksum for image00400000
*** ERROR: Module load completed but symbols could not be loaded for image00400000
*** ERROR: Module load completed but symbols could not be loaded for C:\WINDOWS\system32\xpsp2res.dll
*** ERROR: Module load completed but symbols could not be loaded for C:\IronPythonBug\Delphi\Microsoft.Scripting.Core.dll
*** ERROR: Module load completed but symbols could not be loaded for C:\IronPythonBug\Delphi\Microsoft.Scripting.dll
*** ERROR: Module load completed but symbols could not be loaded for C:\IronPythonBug\Delphi\Microsoft.Dynamic.dll
*** ERROR: Module load completed but symbols could not be loaded for C:\IronPythonBug\Delphi\IronPython.dll
*** ERROR: Module load completed but symbols could not be loaded for C:\IronPythonBug\Delphi\IronPython.Modules.dll
*** WARNING: Unable to verify checksum for C:\WINDOWS\assembly\NativeImages_v2.0.50727_32\System.Xml\773a9786013451d3baaeff003dc4230f\System.Xml.ni.dll
*** WARNING: Unable to verify checksum for C:\WINDOWS\assembly\NativeImages_v2.0.50727_32\System.Configuration\b82c00e2d24305ad6cb08556e3779b75\System.Configuration.ni.dll
*** ERROR: Module load completed but symbols could not be loaded for C:\IronPythonBug\Delphi\Microsoft.Scripting.ExtensionAttribute.dll
*** WARNING: Unable to verify checksum for C:\WINDOWS\assembly\NativeImages_v2.0.50727_32\System\80978a322d7dd39f0a71be1251ae395a\System.ni.dll
*************************************************************************
*** ***
*** ***
*** Your debugger is not using the correct symbols ***
*** ***
*** In order for this command to work properly, your symbol path ***
*** must point to .pdb files that have full type information. ***
*** ***
*** Certain .pdb files (such as the public OS symbols) do not ***
*** contain the required information. Contact the group that ***
*** provided you with these symbols if you need this command to ***
*** work. ***
*** ***
*** Type referenced: kernel32!pNlsUserInfo ***
*** ***
*************************************************************************
*************************************************************************
*** ***
*** ***
*** Your debugger is not using the correct symbols ***
*** ***
*** In order for this command to work properly, your symbol path ***
*** must point to .pdb files that have full type information. ***
*** ***
*** Certain .pdb files (such as the public OS symbols) do not ***
*** contain the required information. Contact the group that ***
*** provided you with these symbols if you need this command to ***
*** work. ***
*** ***
*** Type referenced: kernel32!pNlsUserInfo ***
*** ***
*************************************************************************
FAULTING_IP:
mscorjit!Compiler::FlatFPIsSameAsFloat+7
79098f45 d95dfc fstp dword ptr [ebp-4]
EXCEPTION_RECORD: ffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 79098f45 (mscorjit!Compiler::FlatFPIsSameAsFloat+0x00000007)
ExceptionCode: c0000091
ExceptionFlags: 00000000
NumberParameters: 1
Parameter[0]: 00000000
FAULTING_THREAD: 00000a14
DEFAULT_BUCKET_ID: WRONG_SYMBOLS
PROCESS_NAME: image00400000
ERROR_CODE: (NTSTATUS) 0xc0000091 - {EXCEPTION} Floating-point overflow.
EXCEPTION_CODE: (NTSTATUS) 0xc0000091 - {EXCEPTION} Floating-point overflow.
EXCEPTION_PARAMETER1: 00000000
NTGLOBALFLAG: 2000000
APPLICATION_VERIFIER_FLAGS: 0
MANAGED_STACK:
(TransitionMU)
0012E0F8 02657F5C IronPython!IronPython.Runtime.PythonContext.InitializeSystemState()+0xec
0012E108 026532E7 IronPython!IronPython.Runtime.PythonContext..ctor(Microsoft.Scripting.Runtime.ScriptDomainManager, System.Collections.Generic.IDictionary`2<System.String,System.Object>)+0x247
(TransitionUM)
(TransitionMU)
0012E60C 792EF7C0 mscorlib_ni!System.RuntimeMethodHandle.InvokeConstructor(System.Object[], System.SignatureStruct, System.RuntimeTypeHandle)+0x10
0012E640 792EF55A mscorlib_ni!System.Reflection.RuntimeConstructorInfo.Invoke(System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo)+0xfa
0012E6D0 79288D96 mscorlib_ni!System.RuntimeType.CreateInstanceImpl(System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo, System.Object[])+0x3e6
0012E730 79280E10 mscorlib_ni!System.Activator.CreateInstance(System.Type, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo, System.Object[])+0x70
0012E754 02652BA1 Microsoft_Scripting!Microsoft.Scripting.Runtime.LanguageConfiguration.LoadLanguageContext(Microsoft.Scripting.Runtime.ScriptDomainManager, Boolean ByRef)+0x
LAST_CONTROL_TRANSFER: from 790a40fa to 79098f48
PRIMARY_PROBLEM_CLASS: WRONG_SYMBOLS
BUGCHECK_STR: APPLICATION_FAULT_WRONG_SYMBOLS
STACK_TEXT:
79098f45 mscorjit!Compiler::FlatFPIsSameAsFloat+0x7
FOLLOWUP_IP:
mscorjit!Compiler::FlatFPIsSameAsFloat+7
79098f45 d95dfc fstp dword ptr [ebp-4]
SYMBOL_STACK_INDEX: 0
SYMBOL_NAME: mscorjit!Compiler::FlatFPIsSameAsFloat+7
FOLLOWUP_NAME: MachineOwner
MODULE_NAME: mscorjit
IMAGE_NAME: mscorjit.dll
DEBUG_FLR_IMAGE_TIMESTAMP: 492b8341
STACK_COMMAND: dds 79098f45 ; kb
FAILURE_BUCKET_ID: WRONG_SYMBOLS_c0000091_mscorjit.dll!Compiler::FlatFPIsSameAsFloat
BUCKET_ID: APPLICATION_FAULT_WRONG_SYMBOLS_mscorjit!Compiler::FlatFPIsSameAsFloat+7
Followup: MachineOwner
---------</pre>
<p><i>Floating-point overflow</i> in Delphi when working with foreign .dlls?
Ask uncle Google about it and after a while you find something like <a
href="http://digital.ni.com/public.nsf/allkb/E6A73825E57FCD9F862570DD005E594F">this</a>.
Or, as in my case, Dino sends you the direct link :-)</p>
<p>Another useful commands:</p>
<pre class="prettyprint">
.loadby sos mscorwks
!clrstack -a
!DumpStackObjects
!dumpobj 07d996fc
.load C:\Program Files (x86)\Microsoft Silverlight\3.0.40818.0\sos.dll
.load C:\Windows\Microsoft.NET\Framework\v2.0.50727\SOS.dll
.sympath SRV*c:\dos\DebugSymbols*http://msdl.microsoft.com/download/symbols
</pre>
<p>To recap it:</p>
<p><b>If you have a problem with your program, run it in WinDbg and <span
class="tt">!analyze -v</span> it.</b> WinDbg tells you a lot of useful
information which you can utilize and find a solution for your problem.</p>Lukáš Čenovskýhttp://www.blogger.com/profile/14627955690950344876noreply@blogger.com1tag:blogger.com,1999:blog-1384551653506604356.post-11972736375370517512009-11-22T20:28:00.028+01:002010-10-06T23:26:41.356+02:00WCF Service in pure IronPython with config file<p>I was wrong when I wrote in the <a
href="http://gui-at.blogspot.com/2009/11/wcf-service-in-pure-ironpython.html">last
post</a> that the IronPython service cannot be saved into an assembly. It can.
Which opens a way to use <i>.config</i> file to configure the service.</p>
<p>This is a simple config file for the service:</p>
<p><a href="http://gui-at.cendaweb.cz/2009/11/ConfigService.exe.config">ConfigService.exe.config</a></p>
<pre class="prettyprint">
<?xml version="1.0"?>
<configuration>
<system.serviceModel>
<services>
<service name="ConfigService.myService">
<host>
<baseAddresses>
<add baseAddress="http://localhost:9000/myWcfService"/>
</baseAddresses>
</host>
<endpoint address=""
binding="basicHttpBinding"
contract="TestServiceInterface.ImyService"/>
</service>
</services>
</system.serviceModel>
</configuration>
</pre>
<p>The <a
href="http://gui-at.cendaweb.cz/2009/11/TestServiceInterface.py">interface</a>
is the same as in the previous version. The only difference in the service to
the previous version is in the <span class="tt"> ServiceHost</span>
initialization - we omit the service configuration parameters because they are
in the <i>.config</i> file. I also changed the clr namespace:</p>
<p><a href="http://gui-at.cendaweb.cz/2009/11/ConfigService.py">ConfigService.py</a></p>
<pre class="prettyprint">
import clr
import clrtype
clr.AddReference('System.ServiceModel')
from TestServiceInterface import ImyService
from System import Console, Uri
from System.ServiceModel import (ServiceHost,
BasicHttpBinding, ServiceBehaviorAttribute,
InstanceContextMode)
class myService(ImyService):
__metaclass__ = clrtype.ClrClass
_clrnamespace = "ConfigService"
_clrclassattribs = [ServiceBehaviorAttribute]
def GetData(self, value):
return "IronPython config service: You entered: %s" % value
sh = ServiceHost(myService)
sh.Open()
Console.WriteLine("Press <enter> to terminate\n")
Console.ReadLine()
sh.Close()
</pre>
<p>If you want to run this script, you must save the <a
href="http://gui-at.cendaweb.cz/2009/11/ConfigService.exe.config">ConfigService.exe.config</a>
as <span class="tt">ipy.exe.config</span> to the folder with the IronPython
interpreter <span class="tt"> ipy.exe</span>.</p>
<p>To save the service as an assembly, run the following command:</p>
<pre class="prettyprint">
C:\IronPython-2.6\ipy.exe C:\IronPython-2.6\Tools\Scripts\pyc.py /out:ConfigService /target:exe /main:ConfigService.py clrtype.py TestServiceInterface.py
</pre>
<p>The <span class="tt">ConfigService.dll</span> and <span
class="tt">ConfigService.exe</span> are created. Add the <a
href="http://gui-at.cendaweb.cz/2009/11/ConfigService.exe.config">ConfigService.exe.config</a>
to the same folder and when you run <span class="tt">ConfigService.exe</span>,
the service starts. Note you also need all IronPython .dlls in the same
folder.</p>
<p>You can adjust the .config file to expose a MEX endpoint (<a
href="http://gui-at.cendaweb.cz/2009/11/ConfigService.mex.exe.config">ConfigService.mex.exe.config</a>)
but I don't see a big point in it because svcutil.exe generates C# or VB code.
Anyway - here are the generated files: <a
href="http://gui-at.cendaweb.cz/2009/11/myService.cs">myService.cs</a>, <a
href="http://gui-at.cendaweb.cz/2009/11/myService.config">myService.config</a></p>
<p>You can run the old <span class="tt">TestClient.py</span> and it will
successfully retrieve value from the service. But the old <span
class="tt">TestClient.py</span> does not use <i>.config</i> file. If we want to
use <i>.config</i> file for the client, we have to rewrite the WCF client.
First, here is the sample client <i>.config</i> file:</p>
<p><a href="http://gui-at.cendaweb.cz/2009/11/ConfigClient.exe.config">ConfigClient.exe.config</a></p>
<pre class="prettyprint">
<?xml version="1.0"?>
<configuration>
<system.serviceModel>
<client>
<endpoint address="http://localhost:9000/myWcfService"
binding="basicHttpBinding"
contract="TestServiceInterface.ImyService"/>
</client>
</system.serviceModel>
</configuration>
</pre>
<p>You can see it is very similar to the <a
href="http://gui-at.cendaweb.cz/2009/11/myService.config">generated one</a>. We
do not specify details of the binding but we specify full name of the contract
interface.</p>
<p>If you check the <a
href="http://gui-at.cendaweb.cz/2009/11/myService.cs">generated client proxy
class</a> by svcutil.exe, you see it is based on <span class="tt">
System.ServiceModel.ClientBase</span> and the interface <span
class="tt">ImyService</span>. There are some empty constructors and all methods
from <span class="tt">ImyService</span> interface return result of the same
method name call on <span class="tt">Channel</span> property. That's why I have
created <span class="tt">WcfClient</span> helper function. The client source
then looks like the following:</p>
<p><a href="http://gui-at.cendaweb.cz/2009/11/ConfigService.py">ConfigService.py</a></p>
<pre class="prettyprint">
import clr
clr.AddReference('System.ServiceModel')
import System.ServiceModel
from TestServiceInterface import ImyService
def WcfClient(interface):
class WcfClientBase(System.ServiceModel.ClientBase[interface]):
def __getattr__(self, name):
# if name is method from interface, return the Channel method
if name in (k[0] for k in interface.emitted_methods.keys()):
return getattr(self.Channel, name)
return WcfClientBase()
wcfcli = WcfClient(ImyService)
print "WCF config client returned:\n%s" % wcfcli.GetData(11)
</pre>
<p>The <span class="tt">WcfClient</span> helper function returns an instance of
class based on <span class="tt"> System.ServiceModel.ClientBase</span>. The
<span class="tt">__getattr__</span> checks if the requested attribute name is
interface method and if so, it returns the Channel's method with the same name.
Which is the same behavior as the generated client proxy class in couple of
lines of code.</p>
<p>To save the client as an assembly, run the following command:</p>
<pre class="prettyprint">
C:\IronPython-2.6\ipy.exe C:\IronPython-2.6\Tools\Scripts\pyc.py /out:ConfigClient /target:exe /main:ConfigClient.py clrtype.py TestServiceInterface.py
</pre>
<p>The <span class="tt">ConfigClient.dll</span> and <span
class="tt">ConfigClient.exe</span> are created. Add the <a
href="http://gui-at.cendaweb.cz/2009/11/ConfigClient.exe.config">ConfigClient.exe.config</a>
to the same folder and when you run <span class="tt">ConfigClient.exe</span>,
the client calls the service.</p>
<p>Having this I think there is only a small step to use the IronPython WCF
services in IIS. Unfortunately, I do not know how to do it...</p>Lukáš Čenovskýhttp://www.blogger.com/profile/14627955690950344876noreply@blogger.com2tag:blogger.com,1999:blog-1384551653506604356.post-46282221848744472252009-11-17T13:07:00.006+01:002010-05-16T19:22:20.677+02:00WCF Service in pure IronPython<p>I wrote about implementing WCF service in IronPython <a
href="http://gui-at.blogspot.com/2009/10/wcf-service-in-ironpython.html">a
couple of weeks ago</a>. Meanwhile I pushed Shri a little bit with the <a
href="http://gui-at.cendaweb.cz/2009/11/clrtype.py">clrtype.py</a> and he has
implemented <span class="tt">ClrInterface</span> metaclass there so we can
create the whole WCF service in IronPython now.</p>
<p>The IronPython interface implementation is straightforward:</p>
<p><a href="http://gui-at.cendaweb.cz/2009/11/TestServiceInterface.py">TestServiceInterface.py</a></p>
<pre class="prettyprint">
import clr
import clrtype
clr.AddReference('System.ServiceModel')
from System.ServiceModel import (
ServiceContractAttribute,
OperationContractAttribute)
OperationContract = clrtype.attribute(
OperationContractAttribute)
class ImyService(object):
__metaclass__ = clrtype.ClrInterface
_clrnamespace = "TestServiceInterface"
_clrclassattribs = [ServiceContractAttribute]
@OperationContract()
@clrtype.accepts(int)
@clrtype.returns(str)
def GetData(self, value):
raise RuntimeError("this should not get called")
</pre>
<p>Also switching from C# interface to IronPython interface is easy:</p>
<p><a href="http://gui-at.cendaweb.cz/2009/11/TestService.py">TestService.py</a></p>
<pre class="prettyprint">
import clr
import clrtype
clr.AddReference('System.ServiceModel')
from TestServiceInterface import ImyService
from System import Console, Uri
from System.ServiceModel import (ServiceHost,
BasicHttpBinding, ServiceBehaviorAttribute,
InstanceContextMode)
class myService(ImyService):
__metaclass__ = clrtype.ClrClass
_clrnamespace = "myWcfService"
_clrclassattribs = [ServiceBehaviorAttribute]
def GetData(self, value):
return "IronPython: You entered: %s" % value
sh = ServiceHost(myService, Uri(
"http://localhost:9000/myWcfService"))
sh.AddServiceEndpoint(clr.GetClrType(ImyService),
BasicHttpBinding(), "")
sh.Open()
Console.WriteLine("Press <enter> to terminate\n")
Console.ReadLine()
sh.Close()
</pre>
<p>Notice that we call <span class="tt">ServiceHost</span> with <span
class="tt">myService</span> which is the type and not the instance of our
service. Because of this, the <span class="tt">ServiceBehavior</span> attribute
does not need to have <i>InstanceContextMode.Single</i> parameter.</p>
<p>Finally, here is the test client:</p>
<p><a href="http://gui-at.cendaweb.cz/2009/11/TestClient.py">TestClient.py</a></p>
<pre class="prettyprint">
import clr
clr.AddReference('System.ServiceModel')
import System.ServiceModel
from TestServiceInterface import ImyService
mycf = System.ServiceModel.ChannelFactory[ImyService](
System.ServiceModel.BasicHttpBinding(),
System.ServiceModel.EndpointAddress(
"http://localhost:9000/myWcfService"))
wcfcli = mycf.CreateChannel()
print "WCF service returned:\n%s" % wcfcli.GetData(11)
</pre>
<p>The disadvantage of having just a single service instance is gone,
<strike>the harder configuration remains</strike>. One new disadvantage can be
<strike>it is not possible (yet) to compile the interface and save it to
disk</strike> nor use it from other .NET languages.</p>
<p><b>Edit 22. 11. 2009:</b> See <a
href="http://gui-at.blogspot.com/2009/11/wcf-service-in-pure-ironpython-with.html">WCF
Service in pure IronPython with config file</a></p>Lukáš Čenovskýhttp://www.blogger.com/profile/14627955690950344876noreply@blogger.com6