Repository navigation
Description
Context 🤔
We have been using the "PETE" Stack for the past 18 Months and it has been a really good experience
for end-users (people using the apps we've built), developers and product owners/clients!
We love Phoenix and Elixir and feel it's been a good choice of framework and programming language.
My/our one "regret" is not pushing for more modularity earlier on in our journey with Phoenix,
which has meant that we have "locked up" most of the useful functionality in the Apps we have built
and thus have to "re-implement" lots of "boilerplate" each time we start a new project. 😞
Example 💊
A good example of the PETE stack is Healthlocker https://github.com/healthlocker/healthlocker
The end-users like using and is much better than anything the NHS had before (for this use-case). 🥇
Everyone who worked on the project can be proud of the work that has been done. 🎉
However if you look at the mix.exs file, you will see that while we used several open source modules created by other people, we did not create any reusable components/packages/modules ourselves ...
That's "OK" because it was not the (stated) "goal" of the project to "create reusable code",
the goal of the project was to create an App that helps people "self-manage their [mental] health" and "effectively communicate with their clinicians".
However it was an aim of the project to be Open Source so that anyone could contribute!
And while we "checked the box" of Open Source by making the project public on GitHub,
we have not had any contributions from anyone outside of the "core" team.
This raises a separate question around what constitutes "success" in Open Source: #66 ...
Our hypothesis is that if we had created Healthlocker as a collection of reusable components which then get assembled into an App, we could have aided the "core" goal of the project because by making reusable components, each time one of those components is reused and improved, the original project receives all the improvements "for free".
By creating Healthlocker as a one "monolithic" repository (AKA "monorepo")
we were able to move fast(er) initially and deliver the App to users/stakeholders quicker ... 🐎
but now when we need to implement Auth, Permissions or Chat in a new project 🆕
we have to do it all "from scratch" ... ⏳
Analogy: Smart Phones 📱
Think about the Smart/Mobile Phone in your hand/pocket, it's made up of millions of components.
The software stack that goes into making the smartphone possible is billions of lines of code.
Each time one of the elements in the software stack (Kernel, APIs, UI Libraries, Build Tools) improves,
everyone in the ecosystem (developers, users, companies) benefits from the improvement.
If the stack of software for the smartphone was a "monorepo" it would have failed long ago;
because it would have become unmaintainable and no individual person would be able to understand it!
Note: For this analogy, focus on the software and ignore the fact that for the most part the individual hardware components (screen, battery, processor, camera etc.) are not upgradeable. 🙄
Lessons Learned 📚
We have all learned a lot from our work over the last few years. A few lessons
- The "flexibility" of "NoSQL" rapidly becomes a burden when data is inconsistent and queries slow!
We felt this pain when we tried to use ElasticSearch as a primary DB, Redis to store user data, or JSON in PostgreSQL which resulted in painfully slow queries! MongoDB was a pig when we used Meteor; they sacrificed reliability for "bigness".
- Don't delay/procrastinate on figuring out the schema for your App.
The sooner you think in "data structures" the better. see: "Make Data Structures" by Richard Feldman learn-elm#120 - Embrace Email as a means of connecting/communicating with and getting feedback from users!
Everyone that has access to the internet has an email address. Android (the most popular operating system by far) requires people to use a GMail account. Email is still a good way of reaching people, even if just to loop them back into the App/service you are building. - Proactively Implement Analytics and capture metrics for every aspect of your app(s)!
Trying to retrospectively apply "GTM" as an afterthought is like using a tennis racket to paddle a canoe upstream! It might "kind work" but you'll ultimately "lose" drowning in data noise!
- Building reusable components/modules/packages might take a little more effort up-front, but it makes life much easier the next time we need to build a similar feature/app and invites other people from the community to give feedback and improvements!
Proposal 💡
Our mission for the rest of 2018 is to create the reusable stack we will be deploying in 2019:
Diagram: https://drive.google.com/file/d/1gfvgK1Mn7EuamHJroS8G7FUqauRgA9EH
Note: the colors of the chevrons are not significant. they are merely to help differentiate between the different elements in the stack. If anyone wants to make this look better, please go for it!
If this feels like "a lot", don't be discouraged or overwhelmed, we already have a massive head start!
A lot of the work on this has already been started and in some cases shipped!
Break it down into "chunks" and start from the bottom: the Append-only Log is the core of all apps.
see: https://github.com/dwyl/phoenix-ecto-append-only-log-example (if you haven't already)
@Danwhy is already making great progress with Append-only Log: https://github.com/dwyl/alog
Both @Cleop and @RobStallion have made great contributions to https://github.com/dwyl/autoform
and we are already actively "dogfooding" elements in our client projects!
We'll build a "minimalist" analytics system "from scratch" on top of alog: https://github.com/dwyl/atm
We have done some work towards this in: https://github.com/dwyl/hits
Auth will need to be assembled from the ground up based on alog, fields and autoform.
So for now, ignore the work that had been previously done.
Once we have the "base" layer of alog, Fields, ATM, Autoform, Auth and Admin, we can build the more "feature rich" tools: Contact and Feedback.
Hopefully this will clarify the "strategic" direction we are taking when building our most recent projects in a modular way.
Name? 💭
For now, we don't have a name for the collection of modules/packages.
Open to suggestions as to what we should call this collection of modules.
I'd love to call this the "Obvious Stack" because most of these things are obviously useful to most apps we have (already) built and will give us a massive head start for anything we build in future.
Considering "repurposing" https://github.com/dwyl/abase as it was the "spiritual grandfather" of this.
"aBase" was built and used used for a couple of Node.js Apps in 2016/17 but not developed further, as transitioned to Phoenix in early 2017.
It's not the best name (mea culpa), but at least it conveys the meaning of what we are building.
Feedback / Thoughts / Questions / Discussion ? 💬
As always your feedback is very much invited/requested/appreciated!
If you have ideas, thoughts, questions, concerns, please share!!
Thanks! ✨
Related to: dwyl/learn-elm#121 (Creating Elm Packages) ❤️ and dwyl/hq#497 (Learning Goals). 🎯
Required by: dwyl/feedback#96 (Feedback Widget) and dwyl/product-roadmap#7 (Metrics)
Metadata
Metadata
Assignees
Labels
Type
Projects
- StatusShow more project fields🏗 In progress

Activity
iteles commentedon Oct 25, 2018
This. This is a huge lesson from the work we have done over the past 2 years. Our small modules (which we barely use ourselves) attract a fair amount of attention and contributions - the highly modular approach makes a lot of sense.
Having now done this on our latest two client projects, this way of working makes my heart sing ❤️ 🎶 It's better for us, better for our clients and better for the open source community.
Thanks @nelsonic for the proposal diagram too, having it so clearly laid out is extremely useful!
Cleop commentedon Oct 29, 2018
Thanks for writing this up @nelsonic, I'm enjoying contributing to its progress and looking forward to seeing it take shape!
RobStallion commentedon Nov 2, 2018
@nelsonic This is awesome. I agree wholeheartedly. ❤️ ❤️
This point especially rings true for me. Although I did a lot of work with healthlocker, when I have needed to come back to it to make improvements/fix bugs/add a new feature, I find it time consuming and honestly a little daunting. The great thing about the modular approach and components is that when we need to upgrade a component, everywhere that component is used is also updated.
Having to manually find and update everywhere in an application that you have done something specific can be time consuming and error prone.
I really enjoy the modular approach. There a plenty of reasons for this, the top few being that it saves time on a project if we can reuse something other developers have already built (and we can use that time on the projects rather than reinventing the wheel) and code is in smaller, more manageable chunks (and often more robust as many people have thought long and hard about the best way to implement that one thing).
My favourite reason though is that I feel like the code is much more accessible. What I mean by this is that when it comes to big monolithic projects, they could have (and often do have) plenty of great functions and modules that have been built into them but it is hard to know what they will have and where to look for it. With the modular approach I feel it is much easier to find exactly what it is that you are looking for, apply it to your use case and (maybe most importantly) make suggestions for how the code could be improved/enhanced.
SimonLab commentedon Nov 7, 2018
Agree 👍 if we are able to split applications in multiple reusable components, the creation process will be much faster and enjoyable. For example rewriting the authentication logic for each new application is time consuming and tiring!
However I'm wondering if any features can be implemented with reusable components or do we need to have a limit on the kind of projects that can be implemented with modules?
There is a risk to try to adapt the reusable code to match some specific features of a project. What would be an acceptable solution, accept any PRs in the reusable modules? This might pollute the API and the modules might not be as easy to reuse. Fork the modules to make them specific enough for the application? This will break the idea of reusable component.
I had this issues on previous project where small PRs where added to some npm packages to just be able to add/update some features on the project. So to avoid this we want to make sure from the start that all the features can be implemented easily with the components or we need to direct and think of new features based on what the components can do instead of letting the product owner decide/impose what they have in mind.
It's difficult for me to have a precise idea on how everything will work together, but I like this macro vision on how to build our next projects. A simple example which demonstrate how to use the "PETE extended" would help us to see what are the limitations but also confirm our motivations to go in this direction.
nelsonic commentedon Nov 7, 2018
@SimonLab good question(s)!
Provided we have a definition of all types of data in
fieldswe can build anything!(I don't think we need to impose a "limit" yet. Unless you can think of an example ...)
The only risk I see is in defaulting to the "old habits" of building all the functionality into the "Client App" instead of being disciplined and putting the functionality in a reusable package.
We can only build a simple example once the
alogandfieldsare in a useable state.The "simple" example is going to be "ATM" which will be a useable Analytics package we will use to make "Hits" see: dwyl/product-roadmap#7 a "real world" App with real users. From there we can iterate "up the stack".
Yes, we've done things a little bit "in reverse" with our latest client project because
fieldsis not (yet) built so we are not using it (yet), but that will change v. soon.Our first/next step is to make the docs for https://github.com/dwyl/alog super beginner friendly.
We have a few good examples of this in @dwyl e.g: https://github.com/dwyl/hapi-auth-jwt2
But we should not limit our "inspiration" for "good docs" to @dwyl ... we should find the best docs for any Open Source project and match/beat it!
Don't get "hung up" on the details of how the more advanced features like Admin and Contact will work or look. focus on the "core" components at the "base" of the stack for now.
If it's "difficult" to have an idea, simply go look at
We are re-imagining the Web Application Stack from the ground-up to be distributed, fault-tolerant, real-time, offline-first, metrics-driven, GDPR-compliant and fun to use for developers and end-users.
(if that sounds like a lot of "buzzwords", just focus on "Hits" as the example for now...) 😉
Danwhy commentedon Nov 8, 2018
Thanks for this write up @nelsonic, it's really useful to see the full plan of what we're working towards. I've really been enjoying working on the various parts of this so far, and am looking forward to those we haven't got to yet.
The only question I have is around how all of these modules actually fit together. Based on the above, it looks like each element in the stack depends on those below it, but my understanding was that each of these modules would be completely independent of each other; so if somebody wanted to use only the
analyticsmodule, or only theauthmodule, they wouldn't be required to use theappend only logif that wasn't what they needed.You also mentioned repurposing
abase, so would that be the "link" between all of these modules? (So in our client apps we'd essentially just requireabase, which would give us the full functionality of the stack, in a framework that's essentially "Phoenix+", but the modules would also be available individually for other people that didn't want our whole stack)I just want to be sure I know exactly how independent these modules are going to be, to make sure I'm implementing them correctly as we go on.
nelsonic commentedon Nov 8, 2018
@Danwhy thanks for the insightful feedback and clarifying questions. 🥇
Great news that you've been enjoying diving into building the
alogandautoformpieces. 🎉Answers
Each "core" piece should be independently tested and documented.
I consider
alogandfieldsto be the most decoupled "core" piecesand if someone wants to use
alogand/orfieldsindependently they are welcome/encouraged to!The further up the stack the more "coupled" the pieces become to the "core".
But that does not mean we should build everything into a monolith called "starter-kit" that contains everything... because it would rapidly become "bloated"! Instead what we want to do is build each piece as a "lightweight" MVP of the desired functionality so that we can have people in the community contributing to different elements of the stack without necessarily needing to understand the whole thing. (although our mission is to make the Docs "so good a complete beginner can understand!)
We are not going to build
authto be "general purpose" for any Phoenix project; it will have a "hard" dependency onautoformwhich in turn will be dependent onfields. And since our chosen "data model" for all future projects is an "append-only log" (for accountability, analytics and distributed systems), what we are currently calling "alog" (see: dwyl/alog#2 for naming discussion) will also be a "hard" (not optional) dependency.In our "Client" apps there will be no such thing as a locally defined "User Schema".
The concept of a
personwill be pre-defined inauthand if the Client App needs to add a custom field to thepersonschema it will be done using aperson_custom_fields"lookup table". e.g:inserted_atattribute_id(PK)person_id (FK)field_typedataThis will make the framework incredibly flexible and essentially allows people to "drag-and-drop" from a list of "available fields" into a "custom fields" table, thus the "Product Owner" can construct their "App" based on reusable components (think lego bricks!) without having to write a line of code. "Custom Autoforms" will do the "heavy lifting" to render the forms, validate and store the data and
adminwill help the Product Owner view the data as it is submitted.Yes, this will effectively be a way to create a TypeForm/SurveyMonkey/WuFoo with just a few clicks! 😮
It's going to be much easier to explain the reasoning for each element in the "stack" once the need arises. But here's a basic example: ("abase-ic", see what we did there...? 😉)
Example
Imagine you are building a Social Network for Pet Lovers and you need to store specific data for each member of the platform including their pet's name, species, breed and date of birth.
The "base"
personschema will be the "standard" one defined inauth:inserted_atemail_hash(PK)email_encryptedname_encryptedpassword_hashThe
petstable would be:inserted_atperson_idpet_idname_encryptedspeciesbreeddate_of_birth_encryptedThese fields might be rendered on the person's "profile page" as:
A naive way of implementing this "social network for petlovers" would be to add custom fields to the person ... e.g:
That would be a non-relational/denormalised way and would mean that the "outlier" on the network who literally owns a zoo cannot list all their pets ...
As you can tell from the data we are capturing for the
pet, we can easily construct thepetstable usingfields:name_encrypted:Fields.name_encryptedspecies:breed: `Fields.date_of_birth_encrypted:Fields.timestamp_encryptedRepurposing "Abase"?
Yes, having a "Phoenix+" stack (name TBD) which would include "users" (
people),emails_sent,auth,admin,analytics, etc. is the medium-term goal for this quest.Being able to reference one package in the "Client App"
mix.exsdependencies would be slick.We need help on defining the name for this Application Framework "Scaffolding" ...
#HelpWantedSadly, as usual in computer science / software, most of the "good" names are "taken" ...
The only reason I like
abaseis because it describes what it is/does it's the "base" for building other apps, and alphabetically it would appear first (or near the top) of any "list" of frameworks.But I don't think we are "married" to the name at all! If anyone can think of a better name, please do!
tl;dr
Real App Examples for Learning the Stack?
My personal learning objective (see: dwyl/hq#497) in the next 6 months
is to create one fully functional end-to-end demo/example app using the "New Stack" each month.
Culminating in a "curriculum" of "7 Apps in 7 Weeks": (from my personal notebook/journal...)

that anyone can use to learn how to build using the "New Stack" from anywhere!
Then we "just" need to create the learning content for:
And You can begin to see where this is going ... 😉
We will be each be building one or more "example apps" as complete learning resources in the coming weeks/months. And in the course of building these examples, we will evolve the "stack".
Then, armed with all the examples we will be able to build our own App(s) and any "Client Apps" ...
see: https://github.com/dwyl/start-here#what 😮
nelsonic commentedon Jul 7, 2023
The next element in the stack is Logging. 🪵
So that's what I'm working on now. 🧑💻
That way I can re-use it in all the other projects ASAP. ⏳
nelsonic commentedon Aug 6, 2026
Related: #172