The Data Paradigm

Apps generate data. In some way the app will generate some data. It can be saving user preferences, sending info through some API, telemetry of Operation System (OS) and in the traditional way: saving relational data in a database.

Earlies in 2000’s was normal a software developer thinking about a software keeping in mind what would be the database and the data model. Even using Object Orientated (OO) languages, a lot of professional had Entity Relationship Diagram (ERD or ER) to design the software.

Going back in time a little, we can see basically the same approach with the Delphi 7 (or VB6) developers. In the 90’s developers had the dBASE file system (DBF), where each file is a group of information, or a table if you prefer.  Looking over this architecture, we have something like this:

Usually in this approach we had two disadvantages: 1 – that files are not able to scale to big sizes (and bigger the file became, more extra routines to redo the index was necessary); 2 – the data (and the software) works local only.

And to solve these problems, the dev guys adopted database management system (DBMS), where it is possible have a centralized database (in this case, application can keep local or even running in another server):

If you pay attention you will realize that we have a unique source of information. This way we can have multi-users, even multi-software, using the same data source. And now a days it is large adopted for small/medium softwares.

Another good point of this approach is ACID (atomicity, consistency, isolation, and durability).

Now we can rise some interesting concepts:

  • Data: It is part of the information. In relational database we create a data structure that relate among themselves to build information.
  • Information: We can aggregate data to build info. This way we can process data in a several ways to provide info.
  • Knowledge: It is a state where info has meaning. A good information can able user to make strategic decisions and operations (this user can be even an Artificial Intelligence (AI)).
  • Source of truth: It is a place where I can find data with a defined and desirable structure.

But if you have multiple read/write access to the database (for users and softwares) it will increase. And different softwares will evolve to have its own necessities (and “data needs”). How to scale that?

An easy way to deploy and scale by adding commodity hardware. Besides, DBMS usually have ways to separate different data context (like schemas,  tablespaces, whatever).

But until when will be manageable the costs of hardware increase and the internal organizations of the database?

Scale at this point can not be just focused in the database, because now we are able to realize data is just a state of the information I’ll need. So it can be write in different ways to serve my desires.

Lets try architect a solution thinking in software platform solution:

Clients, in this context, can be different softwares accessing a single service point. This service will process the input and persist in separated data repositories, each one with its own data structure to serve different purposes.

Now we achieve some important points of software development, such as: single responsibility, isolation, encapsulation and so on.

I can scale exactly the piece that I need, for instance the memory of the full-text server.

But it came with side affects: We lost ACID. Keep the sync among the data will be a headache.

To envolve this we can adopt some patterns, such as pipes and filters, for example:

Now we are able to have a unique data provider, which will use high CPU levels of processing to retrieve and transform data in the info requested.

How we are constantly transforming and processing the data, those tasks will consume a lot of CPU and time. Strategies based in cache, commands and queries should be considered.

Just to enlighten and have good examples of those considerations, for cache we can have Memcached or Redis, and to commands and queries we can adopt CQRS pattern.

Before evolving and talking more about the considerations, lets compare all of this with our human life: We constantly receiving information, events and expressing answers, answering requests and giving opinions in different ways.

We can read the news or listem it form a radio. We can use it talking with somebody or writing the same info in a formal way to a paper.

Human has a “cache memory”, like remember a phone number and dial it immediately. Long term memory, like friends name. Other repositories, like a notebook with notes from graduation class. Evolved systems will adopting similar practices.

And again: If you pay attention, database is not so important, is just one piece to store data in a structure that you will need for a determined purpose. That seems crazy, but makes sense when you think that you have a date that will be transformed and stored in different ways to answer different needs.

For instance, a list of desirable products of an ecommerce: You can cache it to answer the web portal efficiently. You will store it in a relational database to long term. You will “decompose” the info in a analytic database.  And finally index it in a full-text mechanism.

That became evident again when you think about microservices, where each one will have you own data bases/data mechanisms.

There is no definitive solution because each system has its own design, and evolve in different ways. But certainly there are common problems to solve, and some practices and patterns can help to solve it. Such as transactions.

We would be able to combine good database technologies with good queue technologies:

The database of the service will handle with ACID and Transactions operations. Some collector from database (or flow to replicate the data, like Oracle GoldenGate) can delivery the info for a queue (Kafka, RabbitMQ for instance) and it handle with the data stream. Full-text search index can be an Elastic Serach, MongoDb, RavenDb or other NoSQL database.

Sure, it can go on and on. But for now, I hope all of this opened you mind far beyond from the monolithic traditional 3-layer architecture software base (Client+Service+Databse… all centralized). So, be prepared for huge companies with amazing architectures to accommodate big systems, or to scale up you software when it needs.

I hope you enjoyed it.

Spaki.

With more than 15 years of experience developing softwares and technologies, talking about startups, trends and innovation, today my work is focused to be CTO, Software Architect, Technical Speaker, Technical Consultant and Entrepreneur.

From Brazil, currently lives in Portugal working at https://www.farfetch.com as Software Architect, besides to keep projects in Brazil, like http://www.almocando.com.br/

Share

698 thoughts on “The Data Paradigm”

  1. Pingback: Google
  2. Pingback: Google
  3. Pingback: transactions
  4. Pingback: sensitizer cream
  5. Pingback: cheap deals online
  6. Pingback: Print
  7. Pingback: termeh
  8. Pingback: 脱毛
  9. Pingback: nebbia leginy
  10. Pingback: گوگل adwords
  11. Pingback: گوگل adwords
  12. Pingback: novin
  13. Pingback: persian tar
  14. Pingback: ارور 1009
  15. Pingback: resume sample
  16. Pingback: داهوا
  17. Pingback: گوگل adwords
  18. Pingback: domain auction
  19. Pingback: سیستم cctv
  20. Pingback: خرید vpn
  21. Pingback: خرید vpn
  22. Pingback: خرید vpn
  23. Pingback: سیستم cctv
  24. Pingback: سیستم cctv
  25. Pingback: سیستم cctv
  26. Pingback: zand game
  27. Pingback: خرید طلا
  28. Pingback: IPhone Wholesale
  29. Pingback: سیستم cctv
  30. Pingback: خرید vpn
  31. Pingback: خرید طلا
  32. Pingback: gratis inserieren
  33. Pingback: خرید vpn
  34. Pingback: nipple tassels
  35. Pingback: سیستم cctv
  36. Pingback: donna vinci
  37. Pingback: drivers hp
  38. Pingback: best vibrator
  39. Pingback: blanket
  40. Pingback: health
  41. Pingback: credit report
  42. Pingback: خرید طلا
  43. Pingback: rabbit vibrator
  44. Pingback: small dick
  45. Pingback: کفسابی
  46. Pingback: sex when sick
  47. Pingback: amber adam eve
  48. Pingback: سیستم cctv
  49. Pingback: mini vibrator
  50. Pingback: خرید vpn
  51. Pingback: خرید طلا
  52. Pingback: using dildo
  53. Pingback: huge dildos
  54. Pingback: felnőttképzés
  55. Pingback: using a penis pump
  56. Pingback: سیستم cctv
  57. Pingback: article
  58. Pingback: Cheapest holiday
  59. Pingback: female sex toys
  60. Pingback: womanizer review
  61. Pingback: realistic cock
  62. Pingback: masturbation
  63. Pingback: lube sex
  64. Pingback: squirt dildo
  65. Pingback: best lube for sex
  66. Pingback: سیستم cctv
  67. Pingback: hp drivers
  68. Pingback: hydroquinone
  69. Pingback: Wire wound filter
  70. Pingback: website traffic
  71. Pingback: penis rings
  72. Pingback: organic pepper
  73. Pingback: خرید vpn
  74. Pingback: mini massager
  75. Pingback: Simeon Yahweh
  76. Pingback: pink vibrator
  77. Pingback: biohealth protein
  78. Pingback: bahis siteleri
  79. Pingback: kona coffee beans
  80. Pingback: aglanmak
  81. Pingback: bahis siteleri
  82. Pingback: bet online
  83. Pingback: bahis siteleri
  84. Pingback: luxury sex toys
  85. Pingback: nubby tongue
  86. Pingback: Morgellons Cure
  87. Pingback: g spot stimulation
  88. Pingback: clash royale hack
  89. Pingback: apartamenti varna
  90. Pingback: sex toys
  91. Pingback: PHP Scripts
  92. Pingback: G-Spot Vibrator
  93. Pingback: Uroxe
  94. Pingback: king cock
  95. Pingback: solaray
  96. Pingback: how to use a dildo
  97. Pingback: sex funiya
  98. Pingback: Top Compras
  99. Pingback: matka result
  100. Pingback: jageze
  101. Pingback: dental cosmetics
  102. Pingback: Sterling Silver
  103. Pingback: lender
  104. Pingback: Sales
  105. Pingback: Tatuaje Mano
  106. Pingback: isis pakistan
  107. Pingback: elangqq
  108. Pingback: Tablet Repairs
  109. Pingback: health
  110. Pingback: psychic
  111. Pingback: Instagram unfollow
  112. Pingback: psychic
  113. Pingback: Computer City
  114. Pingback: gps tracker
  115. Pingback: slovar
  116. Pingback: Peruvian Torch
  117. Pingback: image resizer
  118. Pingback: Computer City
  119. Pingback: برش ليزر
  120. Pingback: ceneval egel
  121. Pingback: prayer
  122. Pingback: golf lessons
  123. Pingback: brazilian hair
  124. Pingback: remy hair
  125. Pingback: sexual health
  126. Pingback: Indian hair
  127. Pingback: lesbian sex toys
  128. Pingback: prepper arts
  129. Pingback: mens fashion-wear
  130. Pingback: malaysian hair
  131. Pingback: peruvian hair
  132. Pingback: sexier heat up
  133. Pingback: Blockchain
  134. Pingback: Best Mini Vibrator
  135. Pingback: Dress
  136. Pingback: light novel
  137. Pingback: travel news
  138. Pingback: prazniki segodnya
  139. Pingback: gastric
  140. Pingback: instrutcii
  141. Pingback: abc
  142. Pingback: Atorvastatin
  143. Pingback: Hydroquinone
  144. Pingback: Hydroquinone
  145. Pingback: sportotobet
  146. Pingback: Cryptocurrency
  147. Pingback: pc build guide
  148. Pingback: خرید vpn
  149. Pingback: diy unicorn
  150. Pingback: Las Vegas Escorts
  151. Pingback: clit vacuum
  152. Pingback: web
  153. Pingback: slovari imen
  154. Pingback: Buy generic cialis
  155. Pingback: Cialis online
  156. Pingback: Buy generic cialis
  157. Pingback: Viagra prices
  158. Pingback: SLICE DIAMOND
  159. Pingback: 3some
  160. Pingback: Computers
  161. Pingback: trah_net_621
  162. Pingback: mouth stroker
  163. Pingback: Anal Trainer
  164. Pingback: make money online
  165. Pingback: Squirting dildo
  166. Pingback: What Is Lol
  167. Pingback: dildo use
  168. Pingback: Rabbit Vibrator
  169. Pingback: kegel exercises
  170. Pingback: penis pumps
  171. Pingback: lovehoney
  172. Pingback: wand massager
  173. Pingback: prostate toy
  174. Pingback: film
  175. Pingback: rotating vibrator
  176. Pingback: best sex toy
  177. Pingback: kinky sex toy
  178. Pingback: Lovehoney
  179. Pingback: adam and eve
  180. Pingback: Sex Toys
  181. Pingback: fitness trends
  182. Pingback: mens outlets
  183. Pingback: free photos
  184. Pingback: Cheap cialis
  185. Pingback: pipedream panty
  186. Pingback: Free Super Tips
  187. Pingback: rampant rabbit
  188. Pingback: 17 noyabra 3766
  189. Pingback: g spot stimulation
  190. Pingback: DraftKings NBA
  191. Pingback: last longer in bed
  192. Pingback: Tire Pressure
  193. Pingback: friv
  194. Pingback: Painting Brisbane
  195. Pingback: College Football
  196. Pingback: College Basketball
  197. Pingback: rabbit vibrator
  198. Pingback: Milf porn
  199. Pingback: سكس
  200. Pingback: how to
  201. Pingback: home inspections
  202. Pingback: Sex live show
  203. Pingback: aflamsex
  204. Pingback: Monay conveter
  205. Pingback: rottweiler puppies
  206. Pingback: Live sexy cams
  207. Pingback: Trending hip hop
  208. Pingback: Livesex
  209. Pingback: bestpornopics.info
  210. Pingback: Car lockout
  211. Pingback: Sexy live girls
  212. Pingback: Freexxx
  213. Pingback: Porno videos
  214. Pingback: Sexy shemale show
  215. Pingback: Cheap iphone case
  216. Pingback: Iphone deal
  217. Pingback: best clit vibrator
  218. Pingback: Iphone protector
  219. Pingback: سكس
  220. Pingback: سكس مترجم
  221. Pingback: mobile marketing
  222. Pingback: before having sex
  223. Pingback: finger sex
  224. Pingback: shibari amazon
  225. Pingback: vibrators
  226. Pingback: Buy cialis online
  227. Pingback: سكس
  228. Pingback: سكس مترجم
  229. Pingback: thai lotto world
  230. Pingback: Free MILF Porn
  231. Pingback: see through thong
  232. Pingback: 2018 NFL Schedule
  233. Pingback: Garo Kassabian
  234. Pingback: ryzen
  235. Pingback: D L Hughley
  236. Pingback: defamation lawyer
  237. Pingback: HONDA
  238. Pingback: holiday segodnya
  239. Pingback: ben wa balls
  240. Pingback: facebook
  241. Pingback: best vibrator
  242. Pingback: mp3 songs
  243. Pingback: aflampornhd
  244. Pingback: Musicpleer
  245. Pingback: page address
  246. Pingback: read article
  247. Pingback: go to the page
  248. Pingback: tongue vibrator
  249. Pingback: follow the link
  250. Pingback: Portrait Art
  251. Pingback: find out more
  252. Pingback: go
  253. Pingback: address
  254. Pingback: sex lube
  255. Pingback: luxury vibrator
  256. Pingback: sex toys review
  257. Pingback: girls masturbating
  258. Pingback: djswoope
  259. Pingback: Cracked phone
  260. Pingback: treasurebox nz
  261. Pingback: Best Butt Plug
  262. Pingback: Best Anal Vibrator
  263. Pingback: Best Wand Massager
  264. Pingback: Bullet Vibrator
  265. Pingback: nipple tweezers
  266. Pingback: kegel benefit
  267. Pingback: poker88
  268. Pingback: light up vibrator
  269. Pingback: glass dildo review
  270. Pingback: MILF Porn
  271. Pingback: Bangalore Escorts
  272. Pingback: Kolkata Escorts
  273. Pingback: Goa Escorts
  274. Pingback: Simple Gazebo NZ
  275. Pingback: tubidy mp3
  276. Pingback: mp3juices
  277. Pingback: Cialis generic
  278. Pingback: masturbation toys
  279. Pingback: dildo review
  280. Pingback: OhMiBod Vibrator
  281. Pingback: Cialis coupon
  282. Pingback: writeessay
  283. Pingback: ebooks for resale
  284. Pingback: g-spot vibrator
  285. Pingback: Buy cialis
  286. Pingback: Cialis online
  287. Pingback: adult games review
  288. Pingback: rocks off toy
  289. Pingback: thai netflix
  290. Pingback: magiya
  291. Pingback: iPhone 7 Cable
  292. Pingback: Lenovo
  293. Pingback: blow job tips
  294. Pingback: notehub.org
  295. Pingback: continue
  296. Pingback: notehub.org source
  297. Pingback: pc games download
  298. Pingback: مه پاش
  299. Pingback: notehub.org
  300. Pingback: notehub.org
  301. Pingback: visit the source
  302. Pingback: delux sex toys
  303. Pingback: here
  304. Pingback: prazdnik segodnya
  305. Pingback: link
  306. Pingback: continue
  307. Pingback: notehub.org link
  308. Pingback: nintendope
  309. Pingback: amazon boston
  310. Pingback: Apple sell
  311. Pingback: first time anal
  312. Pingback: Security Firm
  313. Pingback: rotating vibrator
  314. Pingback: male masturbators
  315. Pingback: tongue sex toy
  316. Pingback: auto taxi
  317. Pingback: Rechnungswesen
  318. Pingback: Garden Shed Paint
  319. Pingback: فال
  320. Pingback: interracial porn
  321. Pingback: pure kona coffees
  322. Pingback: Baby Highchair
  323. Pingback: Freemason Ring
  324. Pingback: katt
  325. Pingback: Retort ring
  326. Pingback: aviz
  327. Pingback: rabbit thruster
  328. Pingback: chin dildo
  329. Pingback: Best Magic Wand
  330. Pingback: Penis Pump
  331. Pingback: Best Sex Swing
  332. Pingback: big black strapon
  333. Pingback: Anal Plug Review
  334. Pingback: stimulating cream
  335. Pingback: Best Anal Lube
  336. Pingback: Samsung wholesale

Leave a Reply