Compare Yourself to Who You Were Yesterday

“Will there be a mandatory history course on campus?” asked a 15 year old Josef during a presentation by his high school alumni. During high school, I’ve always found myself failing the subject and having no interest to study history at all, except to retake the exams. Now, I feel like I’m in a different mental capacity, as I finished my Norwegian History course and exam. Although one may credit the better education system in Norway than Indonesia, but having the right attitude towards learning would be more important. I take this moment to ponder and think about what I have done in the past 10 years.

Although I was never behind in my studies, I would still call myself an underachiever. People who knew me as a child would probably assume I’m a studious person, although my parents can attest that this was not the case. I was one of the kid who would just study enough to pass their exams, and do any other things they enjoyed, for me it was computer games, singing, and occasionally playing the piano/keyboard. Fortunately for me, I never thought that I’d be good in any of them, preventing me from getting side-tracked from my studies. At this point, I was just a shy, sometimes goofy kid, who didn’t know better what to do. Images below would show a low-resolution of my goofiness in high school and middle school.

I had no idea what career to pursue, I didn’t really know what major to choose. At the end of my high school years, Electrical Engineering was very popular, I figured I was good with Physics, especially Electricity. This was a recurring theme during this period of my life, following what is perceived to be good, not out of self-motivation but from other people’s point-of-view. If I were to choose another field where I have no fundamental competence to go through the field, I would’ve been in shambles. However, I was glad to have the supportive friends to help me went through college smoothly.

When I graduated, I began to understand what I’m actually good at, solving problems, discussion, and a bit later in my professional experience, present and pitching my ideas. I wouldn’t have expected this development to happen. I had always thought of myself to be a timid person, everyone said I was a nice guy, but I didn’t know why and what kind of person I was. As I gained more experience, had more exposure and had to deal with more people. I knew how to leverage my personality, to be kind but not a pushover, to try and be confident, but not cocky. I could feel I was learning more about myself, and improving little by little, until I find myself blocked from learning.

I’ve gone through my life not realizing I had a childhood trauma that needed to be fixed, up until one point, one issue could trigger my trauma and kept me down. Trauma-healing was an important milestone in my progress, I was able to address my trauma, dismiss the problem before it went too far. I talked to a psychologist and was able to pinpoint my problem, I’ve always put other people as a priority before myself. Now that I’m abroad by myself, I have taken ownership of what I do, what I learn, and what I love, this has been a great healing process for me.

As I’m writing this, I’m rediscovering myself, what my hobbies are, what my interests are, what my goals in live are. I’ve stopped caring too much about what other people think of me. Rediscovering the kid in me, while also acknowledging I have come along way since 10 years ago. I still have to continue reshaping myself, loving myself, and proceed with what I love for the years to come. Perhaps, when you meet me 10 years later, I’ll be yet another person with different energy and outlook in life.

Learning from the Finns and Norwegians

Soon, I’ll finish my second semester in Universitetet i Sørøst-Norge (USN), a part of my two years Erasmus+ program in Smart Systems Integrated Solutions (SSIs), where I study in Aalto (Finland), USN (Norway), and BME (Hungary). Although both Finland and Norway share some similarity, there’s a slight difference in the educational system. Both offer free education up until Master’s for their citizens, while Norway’s universities are also free for foreigners.

In USN, there is a quota of 50% reserved for those with a Bachelor’s degree from Norway. In Aalto, with all students concentrated in Otaniemi, Espoo, we see a more diverse student community and exchange students, while in USN, you don’t see many international students. In both countries and in both cities (Horten and Espoo), virtually everyone speaks English, so there’s never any language barriers, and in case of Norwegians, you can actually pick up some Norwegian words on your first day. Finnish, on the other hand, is much harder.

In both universities, it feels like the curriculum has been polished in a way that it’s beneficial for the students to understand basic concepts and apply them. Proper care has been made by the professors, with practical projects and lab works, feedbacks and report writing skills that may be a good arsenal for our future career. I would highlight Aalto’s close ties with the industry where we can learn practical MEMS design and case studies from actual industry leaders. This was very insightful and provided the necessary motivation for the subsequent studies, e.g. in USN, where we study more practical microfabrication, measurement techniques and multiphysics modelling. In Vestfold, Norway, there are many electronics industry such as GE Vingmed and Kongsberg, leading in acoustics for medical and maritime purposes, respectively.

Education in Norway, or what we went through in USN was nothing to scoff at. It’s the spring semester, we have evenly spread lectures and assignments, so more time to study and also travel. However, the hard part was the exam week, where we sit for four hours in the final exam, which accounts for most or 100% of our grade. They take the exams very seriously in Norway, there are external examinators and papers are handed in anonymously, minimizing bias. Plagiarism and cheating is also punished severely here. In Aalto, we had more and tougher assignments than USN, but they also contribute towards the grade, making the exams less horrifying.

With harder exams and lesser assignments burden than Aalto, I knew that I had to change my strategy for studies. Particularly in the measurements class, after a quick look of the syllabus and the plethora of different measurement techniques, there was no way to study them all in the final week before exam. For this, I decided to frequently take notes and revise them and review with problem sets and discuss them with peers. For multiphysics modelling, we had many maths and physics to go through, deliberate practice and understanding the assignments were crucial to apply the concepts and the design project was also a big plus. All in all, notetaking during lectures and reviewing often was my key strategy and wouldn’t be possible without my iPad that I decided to buy early in the first semester.

As an Indonesian, these two countries’ educational systems were new to me, but in a good way. Higher quality teaching through industrial ties, well-defined curriculum, maintaining a good honesty policy, and a good student culture provides us the tools to network, building up knowledge, and practical skills for our future careers. Electronics industry, specifically MEMS / microsystems industry is thriving in both of these countries. As someone who is passionate in sensing, electronics, and micro/nano fabrication, the studies have been interesting. My SSIs journey has been a joyful ride, and I’m looking forward for my summer internship in Aalto and my next semester in Hungary.

On Returning to Indonesia: A Reflection

Earlier this year, the Indonesian president, Joko Widodo called out a talented data science expert from NU, Ainun Najib to return to Indonesia. This shoutout by the president sparked some conversations regarding whether or not diaspora should return and build back.

Although most people recognize this move as a gimmick, which is quite common for the current government. We need to also recognize that there are some efforts in the Indonesian research / innovation agency to provide livelihood and space to research for these returning individuals. Grants for research is now more transparent and competitive, but after a rough overhaul, the new BRIN agency is expected to have “growing pains” for a while.

However, when we’re talking about combined efforts between government, university, and industry working hand-in-hand for an innovation pipeline, it goes without saying that the innovation framework in Indonesia are not so desirable. There are still plenty of work to be established with common understanding between these stakeholders.

This lead me to ponder and reflect on my position as an Indonesian student abroad. Although currently I don’t have any bargaining chip, but one must stop and think about the future and the trends. I had the privilege to go through tertiary education and now to study micro systems and nanotechnology for Master’s.

My program trains me to be familiar with micro-systems, more commonly known as MEMS, starting from the material sciences fundaments, fabrication process and measurements + characterization, packaging, and system level design. In a way, we’re studying the whole value chain of a MEMS based sensor.

Higher precision inertial sensors for automotive safety features, self-driving cars, or even body movement in VR may be the driving force for the sensing industry. After one year of study, I would say that what I would probably do after graduation is to work somewhere within these value chain. In my case, I am quite leaning towards the design and fabrication of such MEMS sensors.

Now this brings me back to my earlier point, what should be my ideal steps for the future if I’m committing to this career path. For most people, staying in Europe would probably the obvious choice. However, there are many reasons to return to Indonesia: family, food, and comfort among other reasons.

Staying in Europe has its challenges, being in a foreign environment would always put us some steps behind locals with better networks and know-hows. Going back to Indonesia would mean to “wear many hats” e.g., doing research, lobbying our leaders for support, making industrial partners, and finding the right human resources. Although the first three are nothing to scoff at, sometimes in Indonesia, we find it hard to find great people that would work well with us.

For my case, there is also an ethics dilemma. My education is paid by the EU and I am not obliged to return. However, with a working experience background in BPPT (Indonesian research agency, now merged to BRIN), seeing some intricacies for R&D landscape for some strategic projects in Indonesia, this was my main driving force to pursue my masters. Hoping to find a way to better help these sustainable and strategic projects should I return, is mentioned in my motivation letter. This would speak volume about myself, if I choose to abandon without proper reflection.

Therefore, I’m writing this to help articulate my thoughts on how I ponder the changing landscape, before and after I begin my studies. While I was applying, me and many others in my cohort were hoping to learn more in the latter end of the value chain. That is more towards system level designs and hardware/software engineering. We had to learn more MEMS related courses than we initially thought. It was quite perfect for me, since my head of lab (dec.) taught MEMS in University of Indonesia, I could build on top of these existing concepts.

From the study itself, now I’m facing a paradigm shift, from being an “Application Engineer” to “Component Design Engineer”. This obviously requires different skillsets and strengths, which we are addressing step-by-step in the first year. This also corresponds well with the commonly head belief in the European education system, that what we learn here should open more doors and opportunity than closing them. An optimistic side of me is strongly dissident with the commonly held Asian view of being a super-specialist at what you do.

Being a MEMS Design Engineer would mean to incorporate different disciplines, like many other designers in different fields. The goal is to make a well-functional and useful product, and use whatever tools in your arsenal to make it happen. There’s an art to this, even when you’re essentially doing computational extensive workloads, people would develop an intuition as they become more experienced. This is also true for designers in other field!

At this point, I’ve established that there’s a change in my way of thinking through my education and hopefully will also be matched by my skillsets. Surely, a career change should not matter if I were to return and build values back home. An “Application Engineer” may perform better if they are able to observe the problem directly and provide better solution. Although such case would probably be true for a Design Engineer, there’s something more important, the iterative process.

There’s a reason why electronics are blooming in Shenzhen, because that’s a good place to develop and design the actual electronics. Faster turnaround time between iteration saves costs for the industry. Same argument would probably be valid for the semiconductor industry. For high volume products, it would probably make sense to do it the same way, developing in a R&D fab with steps to scale up the production runs.

MEMS industry in a way was built on top of the semiconductor industry. At this point, after kicking out Fairchild Semiconductor during the Soeharto’s regime, having a new semiconductor industry back in Indonesia would probably be a pipe dream. Knowing how the Indonesian government hopes to align with the EV trends by providing the minerals, pretty much seals the deal for any semiconductor opportunities, at least for now.

I’m hopeful that there would probably a time where MEMS fab would be more feasible for smaller players, but as of now, I would probably think it’s better to start building network of “design engineers” than to return to Indonesia with limited research capabilities. Returning is not the only way to bring back value, sharing some experiences from the industry would also help motivate the next generations of students to undertake the same education.

This is how I would justify my perspective change on returning to Indonesia and how my studies and current understanding of global trends affected my judgement. I also recognize that I am writing this reflection with my limited understanding of the industry as a whole and also the political / innovation landscape of both BRIN and the government.

21 April 2022
Borre, Norway

MEMS dan Semester Pertama di Aalto

Sebentar lagi saya akan menyelesaikan semester pertama di Aalto University dalam studi S2 saya di program Smart Systems Integrated Solutions (SSIs) bagian dari Erasmus+. Semester depan saya akan melanjutkan studi di University of South Eastern Norway. Kesamaan Aalto dan USN adalah posisinya sebagai anggota laboratorium mikro-nano di negaranya masing-masing, di mana anggotanya adalah kelompok industri dan universitas. Secara khusus, kami di Aalto University mendapatkan ilmu terkini dari pakar industri di bidang microelectromechanical systems (MEMS) dan dasar-dasar perekayasaan (engineering) untuk bidang teknik elektro, seperti elektronika berkelanjutan (Sustainable Electronics), rancangan untuk reliabilitas (Design for Reliability), material dan integrasi mikrosistem (Materials and Microsystems Integration), dan Translational Engineering Forum.

Produk elektronik terkini seperti telepon seluler pintar (smartphone) adalah akumulasi dari teknologi yang sudah dikembangkan, salah satunya adalah MEMS. Melalui sensor dan aktuator MEMS, kamera, mikrofon, accelerometer, dan gyroscope dapat diproduksi dengan skala besar dan ukuran yang kecil sehingga dapat diintegrasikan dalam satu produk. Saya mendapatkan informasi tentang MEMS saat kuliah di UI, oleh Dr. Agus Tamsir (Alm.) kepala laboratorium elektronika, saat mengambil kelas MEMS yang diampu beliau. Bidang ini sangat menarik dan membuka angan-angan saya untuk belajar lebih lanjut, beliau juga menyampaikan bahwa jarang sekali MEMS diajarkan di level S1, tetapi kami boleh melihat beberapa dasar dan konsep teoritis di balik MEMS. Hal ini sangat berguna untuk membantu transisi saya belajar MEMS di Aalto.

Belajar MEMS di Aalto, saya banyak mendengarkan langsung kuliah tamu dari pakar industri seperti dari Murata Finlandia, VTT, dan Vaisala. Ternyata pengembangan MEMS di Finlandia yang dipimpin oleh kelompok industri mikro-nano ini luar biasa, Namun, pelajaran di sini lebih komprehensif, wajar karena merupakan kelas di level S2, kami diberi tugas dan arahan untuk mencoba merancang desain MEMS accelerometer sederhana, probing exercise, yang di fasilitasi oleh Murata. Saat mempelajari dan merancang perangkat ini, kami pun dipersilakan mengontak langsung dosen tamu tersebut melalui e-mail. Menurut saya, kelas MEMS ini adalah salah satu kelas berkesan, di mana kita bisa belajar dari pemain besar di bidang MEMS, mendapat bimbingan langsung dari mereka, untuk membantu pemahaman konsep ini.

Akhir kata, menurut saya program SSIs terutama kelas di Aalto University sangat menarik. Kalau ada yang tertarik untuk belajar tentang smart systems, ataupun MEMS, saya sangat merekomendasikan program ini. Pengalaman belajar dari pakar, rekan mahasiswa yang telah dipilih secara ketat, dan beasiswa untuk studi di tiga negara merupakan pengalaman sekali seumur hidup.

Depression, Coping Mechanism, and Mental Health

This is a short story on my experience dealing with depression. Please take care of yourself if you’re feeling down before you implode, or explode.

I have a childhood trauma and problem that’s been holding me back for a long time. I have been choosing to ignore it for the past years. I was so good at hiding this, I have maintained a front of being a goody-two-shoes throughout. Persevering, suppressing my emotions, were the ideal solution to me at that time. While I hate to use the term toxic masculinity, I acknowledge that there were notion in traditional masculinity where showing emotion = showing weakness, and this is definitely not true.

One of the reasons that my depression doesn’t seem to affect me in larger part of my student life was because one of my coping mechanism is learning a new hobby, or studying subjects that I love. It may be weird to mention that I may find deriving formulas to be a good way to unwind, a good challenge as an escape from harsh reality. I am not necessarily good at this but I do see this as a fun thing to do, just like how I was very bad at Counter Strike (the online game), but kept on doing it until I was half-decent.

In a traditional Batak culture, I would’ve been the one who would receive all the attention, spoiled rotten, and all that you would expect from a patriarchal culture. However, that was not the case for me and I’ve felt like I wasn’t so loved, unless I’m sick. Although, I would say that I’m probably not necessarily smarter than my sister, but my coping mechanism and my timid personality did help me go through the childhood trauma way better. I could see my sister crumbling in her teenage years, but I had no such breakdown, or so I thought.

Growing up, I was more of like a black sheep, but in another perspective, I was somehow considered the exemplary sibling. Truth was, I’ve never felt loved as I mentioned, except my sick days. Long story short, I graduated from college, started working, I love what I do, but it would’ve been a lie if I said it was an easy work. I could learn a lot from my job and I was able to eventually got admitted to study abroad. Now, I admit, this doesn’t sound like a story of someone who claimed had suffered from depression.

The childhood trauma of not being loved, being abandoned, was actually never healed during my teenage years. I have considered myself to be a helpful person, but when depression starts seeping, I’ve felt so down that I was unable to do anything productive and in times, became destructive to myself, and maybe to others. I imploded from all the pressure that I had to go through, preparation to move to Finland, family drama, and finishing up my work before I leave. I imploded.

My sister was very helpful during my rock bottom, I was able to talk to a therapist, and I was able to pin-point my issue. I was glad in the end that I have addressed my mental health, I was stretching myself way too thin, trying to help everyone and do everything in my power, but I forgot to take care of myself. Fueled by the deep-rooted trauma, feeling unloved, unappreciated, it got the best of me. Although there are still circumstances that I still regret and despise to this day, I am glad that I was able to get help sooner than later.

I had a big responsibility in my head before I go, but I find it okay to admit that I weren’t up to it. However, I find some joy that while I was suffering under this depression, I find solace in learning something new: remote sensing and GIS. I don’t know if I will ever use this thing I did as a hobby that helped me through the depression phase. But, last week I received a notification that my paper, a result of my work throughout the process, was accepted and I can do a small revision for the figure, but otherwise it’s fine. This paper will forever be a monument of how I tried to overcome my depression and it’s okay to have your own coping mechanism.

I know that this will be an on-going process, I will learn to forgive and I think my time away from the problems have given me a chance to re-think and plenty to work with myself. Time will heal everything, and I am certain that living alone will be a good part of my healing process. First and foremost, I will make sure that I am in a good mental health, going forward with my study and research.

This blog post was not written as a motivation of some sort, but was to tell a story of how I am fighting back against my depression, feeling of inadequacies, and how I plan on going forward. If you feel down and it’s not going away, please seek help, it would be a very good idea.

Kondisi Polusi Udara di Indonesia

Di tengah PPKM, kondisi udara di Jakarta tidak lebih baik dari sebelumnya. Bergantinya musim dan berkurangnya hujan menyebabkan partikel-partikel pencemar udara terakumulasi. Sudah banyak studi yang menunjukkan bahwa PM2.5 berbahaya untuk kesehatan masyarakat, apalagi saat adanya pandemi COVID-19 yang menyerang sistem pernapasan. Ada juga studi yang mencoba menghubungkan konsentrasi PM2.5 di area urban berpopulasi tinggi sebagai salah satu mode penularan virus SARS-CoV-2.

Untuk memantau kondisi atmosfer, Uni Eropa melalui misi Copernicus mempunyai satelit Sentinel-5P untuk memantau kondisi atmosfer dan berbagai gas pencemar. Data Sentinel-5P ini juga dipakai oleh ECMWF untuk menghasilkan data model kondisi atmosfer, salah satunya data PM2.5. Model numerik untuk PM2.5 dapat dibuat menggunakan data Aerosol Optical Depth dan data stasiun pemantauan PM2.5.

Sebelumnya di sekitaran Jakarta ada tiga sensor PM2.5, satu milik BMKG di Kemayoran, dua dimiliki oleh Kedubes Amerika Serikat, sekarang Nafas Jakarta merupakan penyedia data PM2.5 terbesar di Jakarta. Inisiatif baik seperti ini, mungkin dapat direplikasi di daerah-daerah lainnya, mungkin di tempat yang rentan terhadap kebakaran lahan gambut di Sumatra maupun Kalimantan.

Gambar 1. ECMWF Copernicus Atmosphere Monitoring Service PM2.5 Data in Indonesia (https://indonesian-atmosphere.herokuapp.com)

Sementara inisiatif urun daya data polusi udara mulai berkembang, saya mencoba membuat sebuah aplikasi sederhana, mengambil data dari ECMWF CAMS dan menampilkan ke sebuah peta sederhana. Saya menambahkan data pembangkit listrik tenaga uap (batu bara) dan data PM2.5. Semoga percobaan singkat ini dapat bermanfaat dan bisa membuka diskursus atau diskusi lebih lanjut tentang polusi udara. Apakah mungkin kita membuat model CAMS untuk Indonesia dengan data PM2.5 di lapangan dan penginderaan jauh dari Sentinel-5P?

Khayalan Mikrocip Bluetooth di Vaksin COVID-19

Belakangan ini banyak hoax tentang adanya mikrocip Bluetooth atau 5G yang disisipkan di vaksin oleh Bill Gates atau pemerintah RRC. Berita ini jelas salah dan hanya bisa dipercaya oleh orang yang tidak mengetahui kondisi teknologi mikrocip, komunikasi radio, dan elektronik pada umumnya. Walaupun dunia komputer dan digital sudah berkembang pesat, namun masih ada batasan-batasan fisika yang belum ditemukan solusinya. Penting bagi semua orang untuk tahu mana ilmu teknik dan mana fiksi sains.

Sebuah perangkat elektronik yang mampu mengirimkan data, setidaknya butuh tiga komponen ini:

  1. Mikrokontroler atau mikroprosesor, sebagai “otak” dari sistem elektronik tersebut
  2. Pemancar/penerima gelombang radio, sebagai “mulut” dan “telinga” dari sistem untuk berkomunikasi dengan sistem lainnya.
  3. Baterai dan atau sumber daya lainnya, sebagai “asupan energi” dari sistem tersebut.
MtM Technology's M905 AoP module Employs Nordic's nRF52832 SoC
Gambar 1. Modul M905, System in Package Bluetooth dengan Antena

Teknologi yang sudah matang dan bisa menggabungkan berbagai mikrocip dalam satu komponen (contohnya mikrokontroler dan pemancar/penerima) adalah System in Package (SiP). Contoh modul SiP Bluetooth: M905 – Smallest Bluetooth low energy SiP(BLE) module with built-in antenna – MtM+ Technology. Ukurannya masih nampak, dan tidak mungkin bisa ditanamkan tanpa sepengetahuan Anda, terlebih lagi lewat jarum suntik.

Baterai masih belum mempunyai terobosan yang signifikan. Setelah penemuan baterai Lithium Ion, belum ada teknologi yang bisa meminiaturisasi baterai sampai tidak kasat oleh mata. Pengisian daya nirkabel (wireless charging) juga masih belum matang, walaupun banyak start-up yang menjanjikan bisa mengirimkan daya listrik melalui gelombang ultrasonik (UBeam), gelombang Wi-Fi (WiGL), dan lain-lain, ini masih jauh dari kenyataan.

Singkat cerita, fiksi sains yang marak di media populer janganlah menjadi acuan bahwa dunia sudah berkembang secara pesat. Cerita elektronika dalam bentuk cair yang dapat menerima daya secara nirkabel, mengirim dan menerima data dari Bluetooth / 5G melalui satelit yang dipasang oleh Bill Gates / elit global / pemerintahan RRC adalah omong kosong belaka. Kalaupun teknologi tersebut sudah mampu dibuat, apakah bisa diproduksi secara masif? Jika tidak, apa yang menyebabkan Anda yakin bahwa elit global sengaja membuat prototipe yang mungkin senilai milyaran dolar dan menyuntikkannya ke Anda?

Using Global Forecast System from THREDDS Data Server or Google Earth Engine with Python

Global Forecast System, as the name implies, is a global weather prediction model produced by the National Centers for Environmental Prediction. Among the many variables available to be used, the most common parameters are temperature, wind speed and direction, precipitation, and relative humidity. GFS provides weather forecast up to 384 hours after the model is run. This model is run 4 times a day, at 00 UTC, 06 UTC, 12 UTC, and 18 UTC. It usually takes around 5 hours for the model to complete.

GFS data can be downloaded directly from the FTP / HTTPS server provided by the National Oceanic and Atmospheric Administration, for free, since it’s funded by American taxpayers. However, manually downloading a whole (global) file just to work with a subset of data is not ideal, this is why I would recommend using one of the two options I will be explaining here, Google Earth Engine or THREDDS Data Server.

Google Earth Engine is the easiest to work with, you can either use the JavaScript Code Editor or use the Python API with Jupyter Lab environment to begin development. However, there’s an ingestion delay required to convert the GFS in GRIB-2 format to a raster image in the Google Earth Engine platform, if you need the data as soon as it’s generated, you might want to look somewhere else.

Thematic Real-time Environmental Distributed Data Services (THREDDS) Data Server is a way to distribute environmental data for research use. GFS data in its original GRIB-2 format can also be accessed through this THREDDS Data Server (TDS). The data is usually available in the TDS Catalog within 5 hours of the model generation time. TDS has a NetCDF subset service, that allows us to download a specific area of interest (using a rectangular bounding box), variables of interest, and specify the time range (or single time reference).

Both access can be done with Python, using these modules:

  1. earthengine-api (recommended install: geemap, eemont)
  2. siphon (TDS Catalog API)
  3. xarray (ndarray to work with NetCDF)
  4. netCDF4 (NetCDF backend on Python)

You can use Anaconda to create an environment with all these modules and the Jupyter Lab environment:

$ conda create -n geo python=3.9
$ conda activate geo
$ conda install -c conda-forge geemap eemont siphon xarray[complete] netCDF4

Once you’re done, you can use my Notebook example on my GitHub to begin how to access the Global Forecast System on Google Earth Engine:

https://github.com/josefmtd/GeospatialPython/tree/main/GlobalForecastSystem

Computing Weather Data with Python: NetCDF4 and Xarray

When I’m building a system that’s able to compute the daily Canadian Forest Fire Weather Indices, I began with the Google Earth Engine platform. It is capable of doing raster computation at a scale by harnessing the capabilities of the Google Cloud computing platform. Global datasets are updated daily on the Google Earth Engine platform.

In many cases, you can also download the files directly from the data source providers, like ECMWF with their ERA5 product or the GloH2O with their MSWX and MSWEP products. MSWX and MSWEP near real-time data is provided for free (time delay < 1 day) and ERA5T products can be accessible with a lag of 5 days. MSWX and MSWEP can be accessed through Google Drive link, while ERA5T data can be accessed through the Climate Data Store (CDS) API.

To prepare a development environment, I am using Python 3.9 that is provided by an Anaconda environment. I am using Miniconda as a Package Manager / Environment Manager.

$ conda create -n fwi-xarray python=3.9
$ conda activate fwi-xarray
$ conda install -c conda-forge jupyterlab numpy xarray[complete] netcdf4

Once I have my environment, I began writing code for Fine Fuel Moisture Code. I created a FineFuelMoistureCode class to help calculate the FFMC.

import xarray as xr
import numpy as np

class FineFuelMoistureCode:
    """
    Fine Fuel Moisture Code Calculation based on the
    Canadian Forest Fire Weather Index System using xarray
    """
    def __init__(self, temp, rhum, wind, rain, ffmc_prev):
        self.temp = temp
        self.rhum = rhum
        self.wind = wind
        self.rain = rain

        if type(ffmc_prev) == float or type(ffmc_prev) == int:
            self.ffmc_prev = rain.where(rain == ffmc_prev, \
                other = ffmc_prev)
        else:
            self.ffmc_prev = ffmc_prev

    def compute(self):
        """
        Computes the FFMC through Raining Phase 
        and Drying Phase
        
        Returns
        -------
        ffmc : xarray.DataArray
            today's FFMC
        """
        self.raining_phase()
        self.drying_phase()
        return self.ffmc
    
    def __moisture_content(self, ffmc):
        return 147.2 * (101.0 - ffmc) / (59.5 + ffmc)

    def __moisture_code(self, moisture):
        return 59.5 * (250.0 - moisture) / (147.2 + moisture)

    def __rain_normal(self, r_f, m_o):
        delta_m = (42.5 * np.exp(-100.0 / (251.0 - m_o)) * \
            (1 - np.exp(-6.93 / r_f))) * r_f
        return m_o + delta_m

    def __rain_compensation(self, r_f, m_o):    
        mr = self.__rain_normal(r_f, m_o)
        corrective = 0.0015 * (m_o - 150.0) ** 2 * r_f **0.5
        return mr + corrective

    def __no_rain(self, rain, m_o):
        return m_o + 0.0 * rain

    def raining_phase(self):
        """
        Moisture change due to rain from past 24 hours
        """
        # Moisture content before rain
        m_o = xr.apply_ufunc(self.__moisture_content, self.ffmc_prev)

        # Calculate effective rain due to canopy
        no_rain = self.rain.where(self.rain <= 0.5)
        effective_rain = xr.apply_ufunc(lambda x: x - 0.5, \
            self.rain.where(self.rain > 0.5))

        # Use corrective equation for high moisture content
        compensation = m_o.where(m_o > 150.0)
        normal = m_o.where(m_o <= 150.0)

        mo_rc = xr.apply_ufunc(self.__rain_compensation, \
            effective_rain, compensation)
        mo_r = xr.apply_ufunc(self.__rain_normal, \
            effective_rain, normal)
        mo = xr.apply_ufunc(self.__no_rain, no_rain, m_o)

        self.mr = mo_r.fillna(0) + mo_rc.fillna(0) + mo.fillna(0)
        self.mr.rename('moisture_after_rain')

    def __drying(self, mr, E_d, k_d):
        return E_d + (mr - E_d) / 10 ** k_d

    def __wetting(self, mr, E_w, k_w):
        return E_w - (E_w - mr) / 10 ** k_w

    def drying_phase(self):
        """
        Moisture change due to drying phase from noon to afternoon
        """        
        # Equilibrium moisture content for drying and wetting phase
        E_d = 0.942 * self.rhum ** 0.679 + \
            11.0 * np.exp((self.rhum - 100.0) / 10) + \
            0.18 * (21.1 - self.temp) * (1 - np.exp(-0.115 * self.rhum))
        E_w = 0.618 * self.rhum ** 0.753 + \
            10.0 * np.exp((self.rhum - 100.0) / 10) + \
            0.18 * (21.1 - self.temp) * (1 - np.exp(-0.115 * self.rhum))
        
        # Calculate the log drying/wetting rate
        k_1 = 0.424 * (1 - ((100.0 - self.rhum) / 100.0) ** 1.7) + \
            0.0694 * self.wind ** 0.5 * \
            (1 - ((100.0 - self.rhum) / 100.0) ** 8)
        k_0 = 0.424 * (1 - ((100.0 - self.rhum) / 100.0) ** 1.7) + \
            0.0694 * self.wind ** 0.5 * \
            (1 - (self.rhum/100) ** 8)
        k_d = k_0 * 0.581 * np.exp(0.0365 * self.temp)
        k_w = k_1 * 0.581 * np.exp(0.0365 * self.temp)

        # Wetting and drying conditions
        drying = self.mr.where(self.mr > E_d)
        wetting = self.mr.where(self.mr < E_w)
        no_change = self.mr.where((self.mr >= E_w) & (self.mr <= E_d))

        # Moisture content after drying
        m_d = xr.apply_ufunc(self.__drying, drying, E_d, k_d)
        m_w = xr.apply_ufunc(self.__wetting, wetting, E_w, k_w)
        m = m_d.fillna(0) + m_w.fillna(0) + no_change.fillna(0)
                                        
        # Calculate Fine Fuel Moisture Code
        self.ffmc = xr.apply_ufunc(self.__moisture_code, m)
        self.ffmc = self.ffmc.rename('fine_fuel_moisture_code')

By separating the raining phase and the drying phase, I was able to create a more readable code, at least for me. I’ll share another snippet from my final code once I have it all cleaned up and fixed. But, when I had finished coding, I found out that there’s already a submodule in xclim to calculate Fire Weather Indices!

# temp, rhum, wind, rain is a DataArray
# for daily weather measurements

ffmc_prev = 85.0
ffmc_calc = FineFuelMoistureCode(temp, rhum, wind, rain, ffmc_prev)
ffmc = ffmc_calc.compute()

However, I think that this code is much simpler for equatorial area and may be easier to implement for daily FWI calculation.

Open Weather Model Datasets

While building a Fire Weather Index calculation system as an input for the Indonesian Fire Danger Rating System (Ina-FDRS), I’ve been searching for an open dataset that can be used to calculate the daily fire weather indices. The previous version of the Ina-FDRS uses the ERA5 real-time dataset from the ECMWF. This real-time dataset is not free, however the ERA5T with a 5 days delay is available for free through the Climate Data Store (CDS) API. Some of the dataset in the ERA5 family is also available on Google Earth Engine.

Since ERA5 requires a license to obtain real-time data, the next step is to find another dataset. NOAA’s Global Forecast System is freely available, since it’s funded by US tax dollars. This dataset can also be accessed freely on Google Earth Engine and updated daily in the platform. The current Fire Weather Index calculator that I developed is using GFS data as an input alongside the rain data from GSMaP.

While browsing in a forum, I found a new dataset, Multi-Source Weather (MSWX), this model is an operational, high resolution (3-hourly 0.1 degree) with a global coverage from 1979 to 7 months from now. MSWX is divided into four sub-products which are:

  1. Historical record (MSWX – Past)
  2. Near real-time extension (MSWX – NRT)
  3. Medium-range forecast ensemble (MSWX – Mid)
  4. Seasonal forecast (MSWX – Long)

GloH2O, the organization behind MSWX, also provides a global precipitation product named Multi-Source Weighted-Ensemble Precipitation (MSWEP). This product uses rain gauge, satellite, and reanalysis data to obtain precipitation estimates globally.

Both datasets are released under the Creative Commons Attribution – Non Commercial 4.0 International (CC BY-NC 4.0). They’re free to use for non-commercial purposes, by sending a request form, we will receive a link to a Google Drive folder containing the MSWX.

Links:
MSWX – GloH2O
MSWEP – GloH2O