Monday, April 30, 2018

Volunteering Takeaways

I was volunteering at a conference recently. Overall I enjoyed it because it was manual work, and I love manual work (to an extent). I also was able to enter the conference for free and attend many of the talks. I came up with a couple takeaways.

  • There are no limits to how many people will give their time and effort freely, as long as they believe in the cause. The organizers themselves were not really organized at points and its a bit annoying waiting for work. If I decide to volunteer I want to go all out on it so if I'm not given a task to do (and being a one day volunteer its not easy to figure out on my own what to do), I feel like the organizers are not being mindful of my time. If you ask for volunteers, please be organized enough to have plenty of things for them to do. They want to do work!
  •  After completing a task, the volunteers had to go back to the team lead and then that team lead would walk around looking for tasks for us to do. At the end a bunch of us were standing around while the leads team discussed what to do next. Again, its important to be mindful of peoples time.
  • I was having a whole lot of fun moving chairs and tables around. I think back to a book I read called Flow, which talks about how to experience moments. Its all about mental state and If you put yourself in the right state you can really enjoy it, even if the task might seem boring. I was playing around with optimizing the most efficient ways to stack the chairs. How many chairs should I pick up at once? What position should my hands be in? Is it more efficient to create small stacks closer to me and combine them or have one big stack but travel further to the stack? It sounds stupid but its true. If you look for fun & interesting things, you will find them and if you do, you may experience flow, and if you do, you will have enjoyment.
  • I realize that just as I would be really picky in a job, I should be more picky in selecting how I volunteer and which opportunities I want to take on. This was a good experience but I don't think I need to experience it again. I was happy to give my time because the people were actually really great.
  • Going back to the organization. You would think that at these large conferences everything is really structured and people have everything together. However, it was fun and interesting to see the chaos and confusion. People figuring things out as they go. Its quite refreshing really. Its the same as business. You would think large corporations have their shit together. But in reality if you look inside, they are figuring it out just like the rest of us.

Sunday, April 29, 2018

Change The Course

I'm thinking back to a central theme in Slipstream Time Hacking. The idea is that if you move faster (which slows time) and experience more things, you can actually live out more lifetimes (not literally, more in perception). Whereas if you do the same thing over and over again your time speeds up and you wake up one day and wonder what you did for the last decade.

It does have an implication that if you don't experience things differently than you might as well be dead. I don't agree with this because even if you lived the same life your perception at a later age would be different than at a younger age. And if you are quite happy and content with your life then why not keep doing it.

Maybe this is really pointing to the people who are unhappy with their current life or at least a part of it that they wish they could change. But they don't because they are afraid or lazy and life just goes by.

If you are unhappy with today you need to change it today. Otherwise the cycle continues. They key is to make sure that today is different than yesterday. You need to adjust the path. As Slipstream Time Hacking suggests, little changes are good but huge massive change is way better.

And thus we need to actively pursue wormholes, opportunities to 10x the journey. We can't wait for them to appear. Good things happen to those that seize the opportunity. Please don't have Groundhog's Day day, everyday.

Saturday, April 28, 2018

Inbox Zero

One of the most important business skills one should develop is managing emails. Dealing with emails helps you stay organized and focused on the priorities. Every business relies on email for communication to some extent.

Email isn't a bad thing, but another way to think of email is "other people's todo list for you". That means that its important to keep your time in your inbox to a minimum and properly prioritize tasks coming out of it.

The problem with not building a system to manage emails is that you end up doing more work. Have you ever noticed yourself re-reading the same email over and over throughout the day? In order to act on an email you have to first "parse" the email and then act. But often times we fail to act so the email is left in the inbox. Later on you go back to your inbox and in order to figure out what to do, you have to "parse" the email again! Thus, if you use your inbox as a todo list, you may end up parsing the same emails multiple times a day.

So the solution is to act. To make a decision on each email as you parse it. Delegate, Do, or prioritize for later. The GTD philosophy is that if you can do the job in 2 minutes, you should just do it right then and there. This is because the overhead of prioritizing will be more work than just doing it. Thus if you can Delegate or Do it in 2 minutes, just do it. For longer tasks that need prioritization or because it is waiting on something or someone, you need to have a separate system.

For those tasks you can either put it on your separate todo list or mark it on your calendar. But make sure you get the email out of your inbox. You don't want to spend any additional attention on that email until you need to. If you use gmail, check out Boomerang. It lets you set a time for that email to come back to your inbox. It also lets you respond to an email right away but send the email at a specified time. Both these let you act on the email and get it out of your inbox.

Even better than quickly parsing through your emails is to not have to parse through those emails in the first place. Don't receive emails that you don't need to see. Consider unsubscribing to all those ads and newsletters. What's the last time you actually acted on one of them anyway? They all contain an unsubscribe link at the bottom (its required by law). If you want a quick and dirty solution, just create a filter matching for unsubscribe and have them skip the inbox.

Create lots of filters. There's a good chance that you act on emails in the same way. For example, maybe you get an automated monthly report of something. You want to see it when you need it, but its not something that you need to act on right way. You should create a filter to send this filter to a label. That way you can easily view this report on your own schedule. Chances are your won't even bother because its not important. If you really need to read this report the put a reminder on your calendar and allocate time for it. When you pay attention you'll find that a majority of your emails are actually automated shit that you can ignore.

Email is here to stay. Chances are you will have a long career which involve working with email as well as your own personal email account. Start developing awareness and a system for managing it. Happy email == Happy Life!


Thursday, April 26, 2018

How To Trap a Code Monkey

Have you ever been coding and have run up into a problem and gotten stuck. You keep looking for quick fixes to your problem but to no avail. You start getting frustrated. There must be a quick fix along the path that you are on. You keep  moving forward. You just want to finish this last bit of code but its just not working right. That little change ends up taking a lot longer than you had expected and you end up not getting further. Its starting to get late so you are forced to take a break. You go home and have dinner. Then you take a shower. In the shower you have time to ponder it and retrace your steps methodically. You hit an aha moment and go back to your code, this time taking a different path. It works.

That's an example of a "South Indian monkey trap" described in one of my favorite books: Zen and the Art of Motorcycle Maintenance. There is quite a simple trap to catch a monkey. You put some seeds into a hole just big enough for the monkey to fit their hand into. The monkey will put their hand into the hold to grab the seeds. But when they do, their hand becomes too big to be able to pull out of the hole. It demonstrates the concept of "value rigidity" as the monkey is not able to consider giving up what he has already obtained. I forget if the monkey finally gives up the seeds or if it will die their (I would guess it would eventually calm down and let go).

This was the same as our coding trap. We had a solution in mind and when we got stuck. Maybe we tried to move sideways a bit. The correct thing to do is to take a step back and examine the problem but we are so far along the path. The effort to give up what we have and go back seems like too monumental of a task. So we look to go forward down the wrong path, looking for a quick win and we don't find it. In order to solve the problem we have to let go of the seeds. In my example the taking a break and shower lead me to let go and take that step back.

Taking a break is the remedy for when you are stuck. You need to distance yourself from the problem. Completely let go before you can approach it. Otherwise you might still be attached to your current path. You might still have the seeds in your hands.

Don't laugh at the short sightedness of the monkey. In different ways we may ourselves be stuck in  "South Indian monkey traps" and not know it. The monkey trap analogy doesn't just apply to coding. Often times in life we have this value rigidity where we are unable to evaluate our current circumstances due to your previous beliefs. Perhaps we are holding on to a bad career or bad relationship because of mental barriers we have set in place. We won't let ourselves get out of it because we don't want to be perceived in a certain way. Maybe we don't believe we will find something better or maybe we want to hold onto a perception that we haven't "failed" yet.

Pay attention to any stuckness you run into in life. Might you be stuck because you are still holding on to a fistful of seeds?


Wednesday, April 25, 2018

Entrepreneur's Block

Quoting Seth Godin again, "A plumber doesn't get plumbers block, they just do the work". This was actually true for me as well when I was working as a software engineer. There wasn't a day that I didn't push the ball forward. Why? Because it was my job. (yes, I enjoyed a lot of it too and there were plenty of times I pushed the ball even when I was off the job and noone knew, but mainly because it simply was my job). I didn't have trouble with it, sure I got stuck, but I worked through it.

Now that I don't have a job I get to write code for myself (and potential customers), the writer's block is at it again. I've felt the stuckness of not being sure what to work on. Because I've studied this problem before I understand what is going on. I have to break myself out of the mental block and will myself forward.

It seems as you get more and more creative license, the more this writer's block is at work. If you are a low level worker at a call center. You pick up the phone and help the customer. You don't have the freedom in choosing what to do. You may even have a pre written script. As a software engineer you get a little more creative freedom. Although the projects use cases are defined, you have a large say in how the system gets built. But when you are an entrepreneur, especially one starting out without any customers, noone is telling you what to do. You can really do anything. Well, almost anything. And because you have so much freedom, you have so much opportunity for decision deadlock and procrastination because you want to be perfect.

The solution is the same. Treat it like it is your job, and do the work. You must do the work and ship. Progress doesn't happen any other way.

Tuesday, April 24, 2018

Our Needs Are In Conflict

Tim Urban from waitbutwhy.com wrote this great article regarding thinking about choosing a career. It really highlights why choosing a path is difficult because we have so many different needs (5 of them in his example: personal, lifestyle, social, moral, and practical. Often times these needs are in conflict with each other. We want to donate money and help others (social) but this might make it difficult to put food on the table (practical). We want to have lazy sundays and relax and meditate (lifestyle) but also want to get after it and achieve our dreams (personal).

You can't have it every-way and fulfill all you needs at once so you have to choose. Which also means that everyone else has to choose as well. Not everyone can and should make the same decisions. We are not the same people and different needs speak louder than others. So it would be wrong to choose your path based on those around you. It would be wrong to expect others to choose the same path as you.

However, I do notice that a lot of people choose similarly to those that are around them. It makes sense why that is. You are influenced heavily by what you hear and it is very difficult to figure out if it was a path you chose or if it was an idea implanted by those around you. Thus as Tim mentions, it is important to start "de-masking" each need and figure out where it came from and if we should keep it. Did you want to become a VC because you have a friend of a friend who is and he seems to be the most successful? Its ok if that is your reason, you should just understand where you got the yearning from. In the end we don't have needs or ideas in vacuums so I would think that if you drill down far enough, that yearning might have come from someone or something external. If you happen to feel that your needs are different than those around you I would recommend travelling. It was a pleasant experience for me personally to talk with a lot of people who were living a different lifestyle than back at home. Its a good way to gain more perspective.

I think the "yearning octopus" model is a great thing to keep in mind, not only for career, but to understand any or anyone else's life decisions. When you see others doing something different, maybe you won't judge them as harshly because you realize that we all have the same basic needs, its just that they decided to listen to a different "leg" of yearning and make different tradeoffs in life.

Monday, April 23, 2018

What Are Your April 23rd Resolutions?

How are those new years resolutions going? If you are in the majority, you would have given up for a months now. The ones who are still trucking along, good for you! For the rest of you I want to give you an opportunity to re-up your commitment that you made to yourself. The ones you made 4 months ago.

It's OK to fail and recommit. Failing itself is not true failure. The true failure is giving up. I want to encourage you to recommit. If you do this you will build the habit of not giving up. If you fail and recommit over and over again on your deathbed, I'd consider that a success. Just don't give up.

Do not wait until the new year to recommit. If you wait until then you lose that much time, as well as the work you already put in. Today is April 23, so make an April 23 resolution instead of a January 1 resolution (well it's not like I have any daily readership so make a resolution for the day you read this).

Waiting for January 1 is a big mistake. Whenever I hear of people making new years resolutions I know that they will not succeed. Why? Because you have to ask the question: why you are waiting for that day to get started? A true decision means to cut off other possibilities. If you want to quit smoking but will only do so after the calendar flips then you are not truly committed. So if you are waiting for a certain day to get started, without a proper reason, you are not really committed. Don't wait until that date, get started today.

Forget about your past failures. Make a commitment today if you haven't already. Recommit to those failed new years resolution today. The calendar has nothing to do with this.

Sunday, April 22, 2018

Save Money Using Credit Cards

There's a well known person giving financial advice that says that you should avoid credit cards at all costs. The reason he says this is that when some people get credit cards, they immediately buy a ton of things that they can't afford and then get stuck in credit card debt. Thus, he says that credit cards should be avoided and you should immediately cut up the ones you have.

I don't agree with this advice. In fact, credit cards have many good benefits and can even save you money. However, in order to benefit from them, you must be someone who does not do as mentioned above and overspend and go into debt. In that case, you should probably just cut up your credit cards and use cash.

Credit cards give you a couple good benefits. You get very good consumer protections. If a merchant rips you off, you can complain to your credit card and they will often fight on your behalf, often times crediting you back the money. Many credit cards offer you perks such as cash back, airline miles and discounts. Some credit cards cover you for rental car insurance.  I have a credit card (amex I think) that gives me an extra year of manufacturers warranty. These are all really good benefits to have. However, repeating for the third time, if you end up spending more money with credit cards than you would have if you only had cash then you are better off not using a credit card. The value of the extra money you spend will probably be more than the benefits of the credit card.

One more thing. Credit card companies often give you these great benefits because it gives them information on you. They know where you shop and what you buy and yes they do profit off your information. I recently read how Facebook is able to attribute an advertisement (that you saw on facebook) to a sale that you made in a brick and mortar store. This can only be done with either a relationship with the credit card company or the store. Its a bit scary actually so if you are worried about being tracked like this, you might want to consider cash (and not signing up for those store membership cards).

As long as you are not going to overspend, I personally would recommend that you use credit cards. I personally sign up for multiple cash back cards because they often have rotating 5% categories (I have 3 of these, not going to link, just do a search). Thus I can use the card with the most cash back depending on what I am buying. My default card ends up giving me 2% on everything.

These little bonuses add up over time and I really like the consumer protection of these cards. Just make sure you stay disciplined.


Friday, April 20, 2018

Keep it Simple

We have this idea that the more complex something is the better it is. We seek complex ideas and complex solutions. In academia, the more complex your research is the more valued the work is. In engineering, some engineers over engineer solutions to the point that no one can understand the systems they build. Sometimes if we hear advice that is too simple, we might even reject it and look for something more complex. Somehow it seems impossible that a solution can really be that simple.

Keeping things simple is sometimes more difficult than making things complex. Lets take a software application for example. When writing code, if you don't plan out your application, you start adding code here and there as you go. Soon you might have a tangled mess of code. You don't have clear components and boundaries, proper encapsulation. Soon you might make one change which causes another part of the code to break. Although this application looks very complex, it is complex in a bad way. A developer's job is to manage complexity in a program: for example dividing the app into components. Keeping something from getting too complex actually takes more thought and planning. It means thinking about how the project is going to grow and how to break down the complexity into manageable chunks.

Some of the smartest engineers I have met are able to talk about very complex ideas in simple ways. They can interface with both engineers and business people with no technical background. In order for you to be able to do this you need to have a very good grasp of the technical subject. This lets you see it from different people's perspectives. It is said that if someone is not able to explain a complex problem in simple terms, then they don't truly understand it. If you ask someone what a database is and they can't describe it without using the term database in their explanation, that's a bad sign. They should be able to explain to a 4 year old the concept; which probably means making an analogy to something a 4 year old understands: how is a database like an ice cream truck? (I was once asked this question in a Google interview, so at least they realize the importance of this)

Occam's razor tells us that the simplest solutions are the best. Why? because simple solutions are the most direct solutions. When you add more assumptions, you add more possibilities for error. So keep things simple, realize that complex is often times worse (and due to laziness, not smartness)

Thursday, April 19, 2018

The Best Personal Finance Advice

What's the best personal financial advice that I can give you? There's a lot of good tips and tricks: invest in index funds, cut your cable bill, cook more instead of eating out. But the single most important thing you can do is to understand where your money is going. And in order to do this a good idea is to follow the "Getting Things Done" technique of writing shit down.

Ok so you don't literally have to write down your expenses but you need to track them. You don't even need to track every single expense, considering the 80/20 rule, but I'd recommend you do so anyway to get the whole picture. You don't even need to do this for a long time, just long enough. Once you get some period of expenses, you can be pretty certain that your expenses will be relatively the same in the future, unless you decide to change. You can do this in numeral ways. You can use an expense tracker like mint.com, you can use good ol' pen and paper, you can use an app (I'm using AndroMoney).

Once you have a general idea of where your money is going you can decide if your money spending habits match your actual priorities. If they aren't you can make changes. If you notice expenses that you don't really care about, you can make changes. You might even find some services you pay for but aren't using at all. One of the more interesting things that you'll find about tracking your expenses is that your behavior will probably change and you will automatically improve  your spending habits. Why? Because once you start tracking you take ownership for your expenses. When you know you have to log your expense, you will be less likely to make that silly purchase.

Now, this technique is really only for spending habits. Some people may argue that fixing your investments and making money is more important than spending. I disagree with that. While it is still really really important for your overall personal finance, fixing spending is more important than fixing income. Why? Because a dollar earned is not equal to a dollar saved. Your dollar earned is subject to tax, both income and sales tax. This leave about 58 cents per dollar of real earnings. So saving $1 by not spending actually saves you from needing to earn $2. A second issue is that if you don't have financial discipline, even if you earn more you might still be broke. There is a natural tendency for people who earn more to spend more. The world offers unlimited enticing options to put your money and if you haven't sat down and prioritized (is financial freedom and not having to work a 9-5 more important than that new gadget?), its easy to get sucked into the consumerism trap.

Wednesday, April 18, 2018

Death

"Remembering that I'll be dead soon is the most important tool I've ever encountered to help me make the big choices in life. Because almost everything — all external expectations, all pride, all fear of embarrassment or failure - these things just fall away in the face of death, leaving only what is truly important. Remembering that you are going to die, is the best way I know of avoiding the trap of thinking you have something to lose. You are already naked, there is no reason to follow your heart"
"Having lived through it, I can now say this to you with a bit more certainty than when death was a useful but purely intellectual concept: No one wants to die. Even people who want to go to heaven don't want to die to get there. And yet death is the destination we all share. No one has ever escaped it. And that is as it should be, because death is very likely the single best invention of life. It is life's change agent. It clears out the old to make way for the new. Right now the new is you, but someday not too long from now, you will gradually become the old and be cleared away. Sorry to be so dramatic, but it is quite true. Your time is limited, so don't waste it living someone else's life."
- Steve Jobs in his famous  2005 Stanford commencement speech

We don't talk too openly about death in society. Maybe because it scares us. When those that are close to you pass away, you can't help but think about death. But it doesn't necessarily have to be a sad and somber time. It can be very eye opening. As Steve said, death is a good thing because it forces growth and change. We get our lifespan to do what we want to do, no more, no less. Then the next young generation gets their turn. Lets take some time to remember that yes, we will die soon. This is good, imagine the procrastination if we had all the time in the world. Lets make use of the time we have. Rest in peace grandpa. ❤

Tuesday, April 17, 2018

The Power of the Minority

I'm currently reading the book Skin in the Game by Nassim Nicholas Taleb. There is a great section where he talks about minority rule. The intolerant minorities change the world. He gives an example of kosher food.Only .3 percent of the population eats strictly kosher, however, because someone who is kosher is inflexible and cannot eat non-kosher food, whereas everyone else is tolerant and can eat kosher food, many common foods are made kosher by default. Thus the small kosher population has influenced the food choices of the entire population.

Probably a more common life example is going to a party. Many women do not drink beer. But they will drink wine. And men will drink anything. So once you have a number of women, you cannot only serve beer. Thus many times parties default to just wine because everyone will be able to drink it (and keeps it simpler then buying beer AND wine). So again, the minority drives the majority.

Possibly counter intuitive, majorities don't drive change, rather intolerant minorities do. Thus our laws, products, and even languages were probably shaped not for you, but for those with the edge cases.

What do we do? Well there is not too much we can do, because the system optimizes itself in this way to accommodate the intolerant minority. Taleb points out that "an intolerant minority can control and destroy democracy", thus we should be more intolerant with some intolerant minorities. Thus, distinguish if someone at your party really has a valid special request or is just being an intolerant ass and act accordingly.

Monday, April 16, 2018

Share Your Ideas

A common worry of budding entrepreneurs is that they don't want to share their idea with people because they are afraid it might get stolen. Some even go as far as to require an NDA before talking about it. On the surface this makes sense; the more people you share your idea with the more likely it will get stolen. However, in reality, ideas don't really get stolen and this ends up hurting and slowing the business.

I think most entrepreneurs will agree that the more people they talk to about their idea, the more they will get feedback and learn. They increase their chances of finding people that are potential customers or people interested in helping. Not only from that person but possible referrals.

So why should an entrepreneur not be so worried about someone stealing their idea? Well for one, how do you even know it's a good idea? If you haven't validated it,  it may not in fact be a good idea. Even if you do validate it it may not end up being a good idea. Let's pretend it is a really is a good idea.

Well we know that an idea isn't enough, it's all about execution. Its about the work. There's still so much execution to be done. Are a lot of people really waiting for that miracle idea? The people who can execute are already executing and have more ideas than they have time to execute on. Execution itself generates more ideas so they have plenty. The people who can't execute will look at your idea and say "wow what a great idea" and then go back to not executing (it's not the idea that is holding them back).

Let's pretend that someone can execute and the steal your idea. Does it matter? No because you won't actually end up building the exact same product at the end of the day. Your products will still have your personal touches built into them. Even if the core concept is the same, the result, and especially look and feel will probably be drastically different. There are a million and one ways to build a weather or todo-list app.

Finally, many different companies and products can coexist in the same space. If the slice of pie was not big enough to handle multiple products in that space then it probably is a bad idea (or more likely you are just underestimating the size of). Its okay and even good to have competitor products. You can still make plenty of money if you are really solving your customers problems well.

Friday, April 13, 2018

Choose Your Identity Wisely

One of my favorite Paul Graham essays is Keep Your Identity Small. Go ahead and give it a read, its really great.

Did you read it? If you didn't I'll summarize: In order to have a useful conversation about something you cannot identify with it.  This is because once you identify with it the discussion ceases to be around the topic but around you and that makes it personal which makes it hard to talk about. We have feeling to protect. You want to have as many useful conversations as possible in order to learn. Thus the conclusion is to keep your identity small.

But identity can be good both ways. One of the most powerful tools  you can use is identity. It goes with the theory that you are what you believe. One of the famous motivation examples is Roger Bannister running the 4 minute mile. It was widely considered impossible but then he did it. Although the feat was super impressive, the more interesting thing is that after he did it, hundreds of people started doing it, including high school athletes. Thus why belief is so important. So back to identity, this means when you truly identify with something it is very powerful. It shapes your actions, habits, decisions because once you truly believe you are good at X, you actually will become. X can be being good at math, being an athlete, being a great communicator, etc.

That's why I choose to identify with being a fitness beast. This has the opposite effect of what Paul Graham is talking about. Now, like identifying with religion, me choosing to identify with fitness is not specifically to prevent other people from talking me out of going to the gym (because this never actually happens),  but it prevents myself from talking me out of going to the gym. The conversation here is really the self talk in my own head ("oh, it can wait for tomorrow, just relax today", "what's the harm in an extra slice", etc.). By NOT keeping my identity small here it kills that conversation: "Of course I'm hitting the weights today because I'm a f****** beast".

Yes, identity is a very powerful tool and you should use it to your advantage. Whether that is to encourage discussion, or kill it.

Thursday, April 12, 2018

Do The Thing

This is going to sound like common sense but common sense isn't all that common. What should you do to do the thing that you want to do? You should do the thing. Easy right? Well most people actually don't do this.

Instead of doing the thing, its common for people to take a more indirect path. They read books on the thing. They watch videos on the thing. They think about doing the thing and wish about it and dream about it. They talk to their coworkers about the thing. They might take a class on doing the thing or more formal education (which may only have some resemblance to the thing). Maybe its because they believe in order to start the thing they first need some formal certificate from other gatekeepers who have taken that path before.

Ok lets bring this from the abstract to the real. So many people dream about starting a company, building that product. But they do absolutely nothing but learn about it and talk about it. I think it was Peter Levels who made an analogy to swimming. He is swimming for sure. And there are few people who have jumped in and flapping around, trying not to drown. But the far majority of people are up there on the dock discussing swimming: "So are we supposed to turn our arms this way or that way to stay afloat? I've heard of this new technique that is really cool." But doing absolutely nothing. Well, they are reading books about how to become entrepreneurs. Some others are going to school, working on getting an MBA, that way they will have the knowledge and certificate approval to get started.

Now I'm not saying any of this is necessarily bad. Yes there are plenty of people who have taken knowledge from these entrepreneurial self help books and built good companies from them. There are so many people who have successfully built companies after getting an MBA. Many of them met their business partner there or came up with an idea there. But on the other hand, so many people decided not to take this indirect path and decided to jump right in and start doing the thing from the beginning.

You don't need to do all of the auxiliary stuff to do the thing you want to do. You can just do it. You don't know what you need until you do it. When you start doing the thing you will realize what things are truly auxiliary and what things are really necessary. Your stuckness when doing the thing will let you find the right tool to get past it.

Don't start with the path wandering around the thing you want to do. If you want to do the thing. Consider, you know, doing it.

Wednesday, April 11, 2018

Release Your Product by Reducing Complexity

Building a product is incredibly difficult. There are so many complexities and challenges. Most people think of good prfoduct ideas all the time. However thinking of it is one thing, executing on it is something else. Because of the complexity of the task, most people never even get started. This is a real shame.

In order to build a product, you must break things up into manageable tasks. Step one starts with defining an MVP (Minimum Viable Product). An MVP is supposed to be a releasable feature set of a product that

        image from http://amalgamated.io/posts/a-tale-of-two-mvps which states that original graphic is either created by Henrik Kniberg or directly derivative of his work

Releasable is a keyword here because the MVP has to be a product in itself. So like in the image above, you can't just release a wheel*. Instead your MVP must be usable. Its important to put just enough in the MVP. Too many features and you end up putting too much work before you can validate it. Too little and its, well its not viable. 

Remember that an MVP doesn't necessarily mean low quality. Instead of thinking of the MVP as a lower quality pie, you can think of it as a slice of the entire pie. Of course in reality probably less quality because you don't want to spend your time polishing (decorating) that slice of pie. So you want your polishing to be just enough. This will depend on your product (some products, like perhaps creating a "how to make a beautiful website" tutorial, will require more polishing at the start). 

Good. From there even creating an MVP can be quite daunting. You need to break this up into tasks. I personally use Trello to manage my tasks. I start with a card for each high level component. Then I start working on the card I'll remove that card and make more cards for each specific chunk of work. Each card should take about 1 hour to do. If it is more than that it can be reduced. If you find yourself arguing that a card can't be broken down into a smaller chunk and it will take a day to complete, just ask yourself what you plan to do in the next hour. Good, that can be a card. Then the next hour can have a different card. It sounds a little tedious but its important to do this breakdown, otherwise you risk the project being to complex and you not completing the project because of the complexity.

All of this isn't rocket science but you'd be surprised how effective it is. I see too many people not defining MVPs and thinking that they need to spend an entire year before releasing. At the same time so many people fail to get their prototype out while working a job. Life doesn't give you huge chunks of time, so you need to break tasks down so you can tick them off one at a time when you have spare time.

* well, unless your product is a wheel in itself. Either for other companies to turn into cars or for spares.

Tuesday, April 10, 2018

Balance is Overrated

What do people mean when they want work life balance? I think when most people say this it means that they are stressed out at work and would really like time to do things that they enjoy like spend time with their family and pursue personal hobbies. But it turns out that balance in itself doesn't bring happiness. If you read between the lines what people are really saying when is that they are not satisfied with their work.

People don't really want balance. We are brought up thinking that balance is inherently good. But balance doesn't really mean anything. We kind of see it as a little bit of this, a little bit of that, and since we have some of each thing then we have our bases covered. So maybe if we are eating some junk food, if we are "balanced", then we at least will partly be eating well. But its impossible to be perfectly balanced on everything since life has nearly infinite choices for most areas. Instead we really are just taking a subset of our choices and if we differentiate enough then we might call it balanced. For work life balance we might say it is balanced if we do our standard 40 hours and work doesn't cut into our lives past that. Even in that sense, I wouldn't call this balanced because 5 out of 7 days are spent working. Balance is a really unbounded concept.

Studies have show however that people don't really want balance. What they really want is satisfaction. They want deep satisfying work and deep satisfying relationships and deep satisfying other stuff. In order to get this you have to prioritize that work, or that relationship or getting really good at the guitar, or learning 5 different languages. Building deep connections with those around you and expertise is going to take a special commitment. Going deep into building a skillset or working on a relationship or whatever you do is going to require work and commitment. This means that in order to have satisfaction you are going to need to sacrifice balance. This is a good thing, its called prioritizing. It means choosing your calling, choosing what is most important and what you really want. Now you don't have to choose just one thing, but you can't choose everything.

Would you say Michael Jordan was balanced? No, your probably wouldn't because he lived and breathed basketball. Would you say he believes he is living/has lived a very satisfied life? I think most of us would think so. What about Warren Buffet? Ed Sheeren? Oprah? I would guess that any famous person you know of does not have a balanced life in our traditional sense. And that's good because they chose satisfaction.

So don't look for balance but rather find things that you choose to be imbalanced on. If you are looking for work-life balance, a different option would be to look for work that you can commit to.

Monday, April 9, 2018

Monday Fallback Routines

I had a pretty tough day today. I was attempting to do some development work but kept being distracted and had a tough time concentrating. I ran into a ton snags. Some tasks that I had thought were going to be really simple ended up taking a much longer time than expected. In retrospect this isn't bad at all and very expected when coding. However when your expectations are that it is easy and it ends up being hard, it feels like I am behind and didn't get much done.

For those tough days its good to have a fallback. Mine is working out. If I ever feel down in any other area of my life: work is tough, have trouble with a goal, arguments with friends), I can always count on my workout to bring me back into the right mental state. I just finished my workout, pushing myself a bit and I feel good. I don't worry about the issues and stress I had in today's earlier coding session. I can at least say "at least I'm keeping my body in shape and getting stronger." Its great to have this fallback.

Yours may not be working out but maybe you get energy and confidence from some other activity like reading or talking to someone or cleaning or a sport. Its nice to have something as a go to when you have a tough Monday. What do I do if I start slipping on my workouts? I don't know, its never happened and I don't intend to find out.

Sunday, April 8, 2018

Move Your goals Down The Pyramid

Its important to get yourself to the point where you are doing the things that you want to do every single day. You can't take any days off. Why? Because your attitude is different for things that you must do as opposed to the things that you kind of want to do when it is convenient. Lets take a step down Maslow's pyramid. Let's see, at the bottom we have breathing, food, water. Those things are pretty important to you right? You would do them every day even if you didn't really feel like it. Yes yes, for the most part don't have a choice because our physiological system will make you feel it if you don't do these things.

But that's the same way you should approach your goals. If exercise is a priority to you then move it down the pyramid. Just like breathing, you should treat it like you are automatically going to do it every single day. If you don't do it you should feel it. You should feel hurt. Feel disappointed in yourself. You can even set up systems to make it hurt. But that's the attitude you need to have for the things that you really want. If starting that business or getting into that college means the world to you, then it really is just like eating, you will die if you don't do it. Thus you need to treat it that way. If you can't treat it with that much importance, then its a good sign that you don't really want that thing: in that case, life is too short to be chasing things you don't really care about.

When you move things down the pyramid, the you can treat them (and yourself) with a new kind of respect. You can confidently delay requests for your time: "Sorry, I had blocked off that time to (do that thing which is on the line of eating and breating), after I'm done I can help you." You can do this because you've properly set the priority and you know that you MUST do this: its not an option.

Obviously you have to be judicious about what you pick to put there at the bottom of your pyramid. If you pick everything, then you picked nothing. You made no true decision. You must reserve that special place for the thing that is so special to you. You already know what that is.

If you want a special result in something you have to treat it differently. And one of the ways to do that is to put it in the same category as the things you can't live without. That lies at the bottom of your Needs Pyramid.

Saturday, April 7, 2018

There are No Failures

Failures have never been terminal. If you failed, your legacy was never bound to your failure unless you gave up. Mark Cuban had started multiple companies that failed but we don't associate him with failure. We associate him with his many successes that came after. The same with Mark Zuckerberg. He talks about his multiple failed websites before he started Facebook. We don't associate him with his failures either. In fact, nowadays these successful people talk about these failures as a badge of honor.

Although the effect of failure has not changed much over time, I think today our views of failures have changed. And that's a big deal. Because of the power of the internet and our growing access to mainstream successful people, we are able to get a better view of their journey to success. One of the common factors is that none of these successful people got there without failing and learning and improving. Google CEO Sundar Pichai talks about his failures and tells us to wear it as a badge of honor. We all see how young Elon Musk couldn't get a job at Netscape and how Paypal was originally voted as one of the worst business ideas. Even in recent times we've witnessed his many rocket explosions and his challenges with his companies are published everywhere in the media. Since we can follow a long on Facebook and Twitter we see this journey of success differently.

That gives me confidence in the future of humanity. I think our young entrepreneurs will have a lower bar to entry. Much of the difficulties is overcoming the stories we tell ourselves and criticisms from family and peers, possibly even more than the actual work. I'm glad to see people jump right into manifesting their dreams and that gives me confidence to do the same.

Today there is really little stigma around failure. Companies won't look down on someone who tried starting a company, instead they might higher that person because they took risks. If I heard of a peer try out a new experience, rather than laugh at him, I look up to him and am curious about his experience. That experience is valuable. This might be a trend starting mostly in Silicon Valley but I think the trend will continue.

Today we have so much opportunity and many of us have shed the fears that once held many back and decided to take advantage of those opportunities. The time to chase your dreams and risk failure has never been better. What are you afraid of?

Friday, April 6, 2018

Being Securely Prepared

Security is something that is usually thought of after the fact. We get lulled into the idea that if something worked fine the day before then it should be fine tomorrow. So we never take the time to take the steps to deal with the unexpected of tomorrow.

We are in the age of a new kind of cyber crime. In the past, hackers made computer viruses for the sake of it. But now they do so for profit. There are viruses (ransomware) that will encrypt your whole hard drive and then ask for a bitcoin payment in order for you to get your precious photos back. Sensitive information is becoming more and more valuable and is stolen and used for identity theft. Although we shouldn't be paranoid, there are small steps we can take in order to be a bit safer. Here are two of the programs that I use.

Protect your data on your personal computer. I'd recommend VeraCrypt. VeraCrypt is an open source file and drive encryption tool. It works across platforms on Windows, OSX and Linux. You can either encrypt an entire drive or you can create a encrypted container of whatever size you want. You use VeraCrypt to encrypt and decrypt the container. Thus, anything you put into the container will be encrypted. It would make sense to store your sensitive documents in a password encrypted container. I'm sure you'd feel a lot better if you had encrypted your data on your laptop that was stolen or your lost thumb drive. What's also cool is that VeraCrypt offers plausible deniability. This means that noone can prove that there is even an encrypted volume there. All it looks like is random data.

Use a password manager. We live in the age of the internet so you probably have logins for many different sites. Its difficult keeping track of all the different logins and passwords so many people resuse passwords or only make slight changes from site to site. At the very least, the passwords are short and insecure. That's where a password manager comes in. You can login just once with a master password and that will open up access to all the logins to websites. This means that you only need to memorize one secure password. It means you can easily generate long 30 character  random passwords that look like this: WVI&BVCOUATO3p%&Fes*&L^70Lyj2#. This makes it unguessable and if hackers manage to steal passwords for one site, they wouldn't be able to guess your password pattern on another site.

I personally love LastPass.  It comes as browser plugin and can auto fill passwords for many sites. It stores passwords securely in the cloud so that you can access the same passwords on different devices. Although its proprietary, there is a lot of security research around how they have implemented the solution. LastPass does not have access to your master password or your unencrypted stored passwords. Everything is encrypted end to end.

LastPass also has some cool features like being able to share passwords with other LastPass users (useful for family or teams at work), two factor authentication, and even an ability to only share passwords with family members if you die!

The last product I'll recommend today is DropBox. It's likely you've heard of DropBox. It can't be stressed enough how important an offsite backup of your data is. First of all, its important to have any backup of your data because storage drives die (or get stolen) all the time. Backing up to another harddrive is much better so that you can recover your data, but it won't prevent you from disasters such as a fire (or maybe both your drives die or get stolen). That's why its important to have offsite backups such as DropBox. So make sure to backup important documents using something like DropBox. To make this even nicer, you can use VeraCrypt to create an encrypted partition in DropBox so that your data is encrypted in the cloud! How cool is that?

Fyi I'm not paid anything to endorse these products. I'm currently not paid to do anything these days ;) These are just some security related products that I use and I'd like to pass on the info. At the very least, take a moment to think about a few disaster scenarios and a few steps on how to make that better. Just like practicing a fire drill, its important to practice thinking through and preparing for security incidents.

Thursday, April 5, 2018

Your Expectations Are Hurting You

One of the greatest insights I've had is by the great Tony Robbins. We are sad or angry not because what happened to us, but instead, of our expectations of what should happen. On the surface this doesn't seem like such a key insight, but it really does have some profound implications.

Lets understand this with an example. Lets say that I'm at work and my coworker tells me that I am a terrible engineer. I take pride in my work, so I am offended by the statement. I am hurt and sad. Now, a while ago, I would agree that I am sad because of what my coworker did. But, after that key piece of insight, I now know that I am sad because of my expectation of what should happen: "He has no right to treat me like that", "who is he to judge, he's not a good engineer to judge", etc. The expectation is key here because if I thought that this was normal: maybe I am a crappy engineer or maybe this was expected because this particular engineer says this to everyone, then I'd go on with my day because what happened would have aligned with my expectation. But it was the fact that it didn't, that got a reaction out of me.

Now we know that my reaction was based on my expectation. Lets take it one more step. If my expectation was different than what happened, it means that my expectation was different than reality. That's right, I've become sad or angry because I live in a world that is not real and that real action has caused my imaginary world to come crashing down. When you put it this way, it turns what we originally thought as a basic normal action(person said something) / reaction (I reacted to what was said), to me just being a bit cuckoo. You believe that person should not say those things to you but she did! You believe your son should wash the dishes but he didn't! You believe your significant other should have been more thoughtful to you but they didn't! How long are you going to avoid reality?

Once we realize how illogical we are being we can take the necessary steps to address the issue. You have to face up to reality. Accepting reality what is is half the battle. When you do that you no longer blame. Its not your kids fault for not washing the dishes. Its your fault, for expecting the kid to have done so. From there you can calm down, stop blaming, and take action to make it better. Did your kid know that it was his responsibility to do so? Is he rebelling because of some reason? Does he need and can you help him build his discipline? There can be a number of reasons and you can now thoughtfully take ownership and seek ways to make things better.

In order to make things better we need to make a distinction on what is part of our expectation and what is reality. It sounds crazy that we could get really emotionally distraught over our expectations but we do and will continue doing so until we learn to embrace reality.

Wednesday, April 4, 2018

The Games We Play

When we do anything in life, we are in a sense playing a game. Each game can be either infinite or finite. Infinite games are games where there is no ending. Finite games are games where there is an ending. A finite game might be a basketball game: when the 48 minutes are up, the game is over. An infinite game would be family. We are not trying to win each encounter with friends and family, instead we do it for the sake of playing the game and expect it to go on forever. There are some cases, going to work for example, where the game we are playing is not predetermined and we can play them as a finite game or infinite game.

Simon Sinek has done some great research on finite vs. infinite games within companies. He recalls that an Apple exec was shown Microsoft's new Zune and didn't worry about if it was better than the current iPod. He knew that Apple was playing the infinite game and continually make better products. Sometimes they'd have the best product, other times not. But it didn't change what they were trying to do. Google spends much of its resources building infrastructure for the internet and gives out much of it for fee. It knows that the more people who have access to the internet, the more tools there are for small business owners, the more customers Google will have down the line. They are playing the long game. Not surprisingly, companies that play infinite game tend to stay successful and outlast their opponents.

A lot of the difference in results is due to purpose. Finite games are played with the purpose of winning. This means short term thinking, seeking predictability, playing within the bounds. Finite games tend to be more serious. Infinite games are the opposite. The purpose is to keep on playing. That means keeping the game fun, long term thinking, and changing the rules when needed. Unpredictability is embraced. Success looks and feels much different between the two play styles. Success from infinite games is deeper and more meaningful and more likely to succeed.

So its worth while to think about our own games in our personal lives.

Are you going on a diet? That's a finite game. Is your current diet a healthy one and one you want to stick with for your life? That's an infinite game.

Are you working out so you can get that beach body this summer? That's a finite game. Are you working out a couple times a week to maintain your health forever? That's an infinite game.

Are you putting in extra hours to earn that promotion at your new job? That's a finite game. Are you committed learning and improvement, putting in extra time every single day to learn something new? That's an infinite game.

Are you an entrepreneur who has spent the last 6 months building a product, betting everything on your one idea being successful? That's a finite game. Are you an entrepreneur who is willing to consistently put things out there and get feedback and learn from mistakes, getting smarter and better each time? That's an infinite game.

Are you a blogger who will write 10 articles and give up when noone is following you? That's a finite game. Are you a blogger who will write an article a day, forever, in order to improve your writing and help you focus your thoughts? That's an infinite game.

Games seem the same if we just look at a small slice of time. However the way we play them and the length we play them can make a world of difference.

Tuesday, April 3, 2018

Hosts File

When you try to access any website, your computer takes a look at your hosts file before making the request. A hosts file maps a hostname like www.google.com, to an ip address like 172.217.14.68. That ip address is necessary for a computer to access the correct servers and get the web page you are looking for.

A hosts file is a map of a hostname to ipaddress that you can modify yoursel. The ipaddress that is specified in your hosts file takes precedence over DNS lookups(another way in which your computer gets an ipaddress from a hostname). There are some pretty useful uses for this.
  1. Block unwanted connections (ads for example). This file can basically double as an adblocker. The ip address 0.0.0.0 is specifically designed to be an un-routeable address. This means that if you map a hostname to 0.0.0.0 you will not be able to go to that url in your browser. You will get a "This site cannot be reached" message from your browser. This also means that if you are surfing on a site and there are a bunch of requests to ad domains for example, they can be blocked as well with the Hosts file. You can use a currated list to add to your hosts file which blocks a bunch well known ad hosts. It has an equivalent effect of using an adblocker but without needing to install anything.
  2. You can take it further by blocking specific sites that you waste time on. I'm currently trying to be diligent about how I spend my time so I added this to my Hosts file: 
                   0.0.0.0  www.espn.com  espn.com

          I should note that yes, you do have to add subdomains for them to be blocked.

      3. I have my raspberrypi git server running with a static ip so I added this line:

            192.168.0.111 raspberrypi
          This lets me ssh to raspberrypi instead of 192.168.0.111. Its a lot easier to remember and type.

There's many more uses for this but this is what I am using it for for now. It's quite a powerful little tool. So keep it in mind if you are working around domains and ips.

Oh right, to access your hosts file:

Linux: /etc/hosts
Windows: C:\Windows\System32\drivers\etc\hosts
OSX: /private/etc/hosts

Monday, April 2, 2018

Google's Machine Learning Crash Course

Machine Learning is taking over the world. Google is offering a free Machine Learning Crash Course. I finished it over 2 days and enjoyed it a lot; particularly their exercises where you can tweak parameters and see the result as your model is training.

Here are my notes.

ML Concepts


Introduction to ML


Machine learning makes us think in different ways, specifically takes it from a mathematical science to a natural science. Machines can do things that we have no idea how to do ourselves.

Framing


Some terminology.

Supervised ML produces predictions on never-before-seen data (much more common than unsupervised)
Unsupervised ML is like true AI, it can identify patterns without human guidance: for example can analyze a group of data and come up with its own way of classifying them.

An example is a piece of data. Examples have Features which describe that data (name,subject,body text,etc..) The example is Labeled if someone labels it (ex. spam or not spam). If no one (or thing) has labeled that example for us, then it is unlabeled.

A Model is the thing we are trying to create through ML: a map of examples to predicted labels

There are different types of models

Regression model: what's the probability of X? value of y?
Classification: hot dog, not hot dog

Descending into ML


Linear regression is a method for  finding a line that best fits the data points and something you did in high school algebra 1. If you remember, the equation for a line is y=mx+b. Machine learning has the same equation but different terminology.

y' = b + w1x1

y' is the predicted label
b is the bias. In some ML doc, it will be w0
w1 is the weight of feature 1. Weight is the same concept as slope.
x1 is a feature


The Loss of a data point is the difference between that line drawn and that particular data point. You can think of it as a penalty for a bad prediction for that example

L2 Loss is the square of the loss for a given example. If you take the average of the squared loss for all the examples it is called Mean square error (MSE)

Just like you train in the gym with weights and bias. Training a model means figuring out good values for weights and bias. In supervised learning it is all about minimizing loss, also called empirical risk minimization


Reducing Loss


How do we choose the model that minimizes loss? One way to do this is in small steps (Gradient steps) that minimized loss, called Gradient Descent. 

So you pick initial starting values for your model. For linear regression that would be the bias (b) and weight (w1). Then you would run the calculation over each point (or a batch of points) of your dataset to calculate the MSE.

From there you would compute parameter updates: this is done using an algorithm to calculate gradients. Its math but the general intuition of the math isn't really complicated. Remember a partial derivative just tells us how much the result is changing.  A gradient is just the partial derivative for each independent variable in the function. So if your function has x and y. Your gradient has 2 values: derivative of f with respect to x, derivative of f with respect to y. So if we know the change of x and the change of y. Now believe that if we know the change of x and the change of y, we can combine that with where we currently are to calculate the general direction of where we want to go. This is called the Directional derivative.

So we can use this Directional derivative to choose better values for the bias and weight. Then run the calculation again to give better results. Each one of these cycles is one step. When the loss stops changing or changes extremely slow, then the model has converged.

You want to set your Learning Rate or step size to something efficient. Too low and you take take too many steps and learn to slowly, too high and your steps dance around your goal and you may never hit your loss minimum.

Keep in mind that some problems will have more than one minimum so in these problems your starting values are important.

With large datasets, recalculating the MSE for all datapoints is too data intensitve. You can get nearly similar results if  you just do it over one example (Stochastic Gradient Descent or SGD) or a batch of examples (Mini-Batch Gradient Descent). You can think of SGD as just a mini-batch of 1.

steps and batch size are examples of hyperparameters. Parameters to tweak to tune the ML model.

First Steps with TensorFlow


TensorFlow (TF) consists of a hierarchy of tool-kits. TensorFlow itself models Java and the JVM. TensorFlow's file formats called protobufs (similar to Java... well only in analogy), run on a graph runtime (similar to JVM).

This crash course will focus on the high level API: tf.estimator, but you could really do the same thing with TF lower level APIs.
Quick Introduction to pandas
Pandas is a column-oriented data analysis API. The two main data structures are the DataFrame; similar to a relational data table with rows and named columns. A single column is a Series.

Important Note: pandas is not named after the cute bear but rather it comes from the term panel data.



pd.Series(['San Francisco', 'San Jose', 'Sacramento'])
Out[3]:
0    San Francisco
1         San Jose
2       Sacramento
dtype: object

Series can be combined into DataFrame:

city_names = pd.Series(['San Francisco', 'San Jose', 'Sacramento'])
population = pd.Series([852469, 1015785, 485199])

pd.DataFrame({ 'City name': city_names, 'Population': population })



You use pandas to manipulate data. For example: you can add a new column like so:

cities['Is wide and has saint name'] = (cities['Area square miles'] > 50) & cities['City name'].apply(lambda name: name.startswith('San'))
cities
Out[42]:
City namePopulationArea square milesPopulation densityIs named after saint and city area > 50 square milesIs wide and has saint name
0San Francisco85246946.8718187.945381FalseFalse
1San Jose1015785176.535754.177760TrueTrue
2Sacramento48519997.924955.055147FalseFalse

 Each column and row has an index on creation. Reorder columns by reordering the index.

cities.reindex([2, 0, 1])
Out[49]:
City namePopulationArea square milesPopulation densityIs named after saint and city area > 50 square milesIs wide and has saint name
2Sacramento48519997.924955.055147FalseFalse
0San Francisco85246946.8718187.945381FalseFalse
1San Jose1015785176.535754.177760TrueTrue


First Steps with TensorFlow (exercise)


This exercise is an example of building a model to predict housing value based on number of rooms.

To build the model we first need to define feature columns and targets.

In this model we'll do a linear regression using a Gradient Descent (described earlier). A gradient clipping can be added which is just a cap to make sure steps are not too large, causing the descent to fail. It's like training wheels to a bike.

The last step is to create a function that pre processes the data to feed into the LinearRegressor.

After that the model can be trained. By examining predictions over periods of steps we can see the gradual improvements in predictions and tweak our hyper parameters(learning rate, steps, batch size)  to get to our minimum loss defined by RMSE



The straight lines are the lines at each period, ending with the final predictions red line. Each period is better and bettor, ending in a decent RMSE. This is an ideal gradual descent.

In this graph you can see that the RMSE is descending gradually. You don't want it to be too linear because that would be descending too slow. A descent like this is a good sign of a well tuned model.

Synthetic features and outliers


continuing from the previous housing example. A synthetic feature is a feature that is made up of other features. In this case we will create a synthetic feature as the combination of two features: total_rooms and population.

We can use a scatter-plot and histogram to detect outliers

Notice the outliers. These outliers can be clipped by applying min/max functions to our feature.

Generalization


Its a mistake to try and overfit your training data: meaning trying to draw a line that covers all points. While it works for your specific training data, the overfittedness can cause new data to be mislabeled where it would not have been for a less aggressive fit.

One way to prevent this and to test if our model is any good is to take some amount of our dataset to use for training and take a different amount of data from the dataset for testing. Do not test on your training data.

To draw data correctly:
  1. data must be randomly drawn
  2. the distribution is stationary, it doesn't change over time
  3. we always pull from the same distribution
Keep it simple -Occam's razor

Training and Test Sets


Ideally you want a good number of data for test datasets and validation datasets. If your dataset is large you are fine. If it is small then these are in conflict. You trade off confidence in testing vs validating. But if it is small you can do something like cross validation.

Validation


The more rounds of testing on the same test set causes you to overfit to that test set. One way to solve this is to create a Validation Set. Now you train on your training set, then validate on the validation set. You can do many iterations of this. Finally you confirm the results on the test set. If the results on the test set vary from the results on the validation set, then its a sign of overfitting.

Validation Exercise


Do checks on the data to make sure there are no errors. Pay attentions to any data caps, values outside of sane ranges. Doing a plot of the test and validation set samples separately will help you spot errors.

Representation


Getting features from real world data, called Feature Engineering is what ML scientists do most of the time. You can use 1-hot encoding to map a feature to a unique representation. 

Good features are those that have realistic and easy to understand values. Don't use magic numbers for features. They have to appear a reasonable number of times in the data set.

You can use the Binning Trick to create ranges and consider that range (or bin) a feature.

Feature Sets Exercise


You can use feature sets to train your model instead of a single feature. To find features for your feature set, use the corr() method on a dataframe to show a correlation matrix. This shows how much one feature correlates to all the other features. Pick features that are not strongly correlated with each other so that they add independent information.

Feature Crosses


Sometimes the data cannot be fit well by a linear line. Feature Crosses solves this by creating a synthetic feature (by multiplying two more more features) so that it does fit a linear learner. This is important because linear learners scale well to massive data.

Regularization


At some point your model training predictions will get better and better but your general data validation will start to get worse. This is an example of overfitting to the training data.  Regularlization is a strategy to prevent this. Strategies range from stopping early, to penalizing the model complexity.

One way to define model complexity is via weights; smaller is less complex, which is better.

L2 regularization has a model complexity defined as the sum of the squares of the weights.

The goal now is to minimize the combination of the loss AND model complexity rather than just loss.
You can tweak your model by adjusting the lambda: this gives more or less weight to the model complexity. High lambda = simple model but risk of underfit. Low lambda = complex model but risk of overfitting. What you decide depends on your specific data and circumstances.

Logistic Regression


A normal linear regression would give us strange values for a coin flip. Instead we need to use Logistic Regression to give us probability predictions. This is also efficient for large data sets.

Classification


Classification is defining if a puppy is cute or not cute or if something is spam or not.

We use a classification threshold to draw the line of when to label something  as spam or not. There are tradeoffs of Accuracy and Precision if setting the threshold too high or too low.

What metrics can we use to know if our classification is any good?

Accuracy = the number of correct classifications / the total predictions.  We can't just rely on accuracy because of class-imbalanced problems: problems where one of the classifications might be extremely rare compared to the other(s).

Instead of Accuracy, better metrics are:

Precision = true positives / all positive predictions. Only say wolf when we are absolutely sure. Tradeoff is some wolves not predicted. (false negatives).

recall = true positives / all actual positives. Say wolf when there is a rumble in the bushes. Tradeoff is some rumblings end up to not be wolves (false positives)



Receiver Operating Characteristic curve (ROC) - Plots True Positive Rate vs. False Positive rate. Stated another way: if we pick a random positive and random negative, this is  the probability the model ranks them in correct order.

Prediction Bias is the sum of all the things we predict and comparing them to observed. 
Prediction Bias = average of predictions - average of labels in data set.
No prediction bias (0) means average of predictions == average of observations. So if 1% of emails are spam then the average of predictions would be 1% in order for there not to be bias.

There is something wrong with the model if there is bias.

Regularization: Sparsity


Sparse feature crosses significantly increases feature space which leads to RAM usage and overfitting.

L0 regularizing: zero out weights. NP hard so not efficient in practice.
L1 regularization: penalize sum of abs(weights). Encourages many of the useless weights to be exactly 0.

Introduction to Neural Nets


Some datasets are really complex. We'd really like Neural Nets to model it for us.

In order to map a non-linear model a non-linear function is needed. A simple one, ReLU, gives great results. To make a neural net, a Non-linear layer of nodes needs to be added. That makes it non-linear and now more linear layers can be added.

Training Neural Nets


Back propagation - its all about gradients. Each layer can reduces signal vs. noise. Gradients can explode if learning rate is to large. ReLUs can die if the values go into the negative. If that happens change initialization values and start again.

Normalizing feature values helps Neural Net training. Different optimizers, such as Adagrad and Adam can help on convex problems (but not ideal for non-convex problems!).

Dropout means that for a single gradient, a node is randomly taken out of the network. This helps regularization. 

Multi-Class Neural Nets


Sometimes we need to classify to multiple classes; not just cat, not cat. But is it a dog, cat, human, or cow.


With multi-class, single label classification we can use soft-max which just makes sure all classes have a sum of probability == 1. This is similar to logistic regression. Softmax is expensive as the number of classes grows so there is an optimization called candidate sampling so that not all negative samples need to be modeled.

With multi-class, multi-label classification use a one vs-all strategy: there is output layer which has a binary yes/no option for each of our classes.

Embeddings


Embeddings are a tool to map items to vectors. Movie suggestions can't really be embedded on a single line: Someone who wants to watch a childrens movie might also want to watch a blockbuster movie. Thus a 2d embedding is more appropriate with a 2d space of children/adult and blockbuster/arthouse.


How do you get data for movie recommendations? If a user has watched 10 movies you can take 3 to use as labels and use the other 7 for training.

How many embedding dimensions to use in layers? There are tradeoffs between more accuracy and overfitting. A good rrule of thumb: dimentions = 4rth root of(possible values)

ML Engineering


Production ML Systems


ML code is only 5% or less of overall code for ML systems since they do a ton of other things.

Static vs. Dynamic Training


Are you training offline and just once? Or online as data is coming in? What you use depends on if your data will change(trends/seasonalities)

Static vs Dynamic Inference


Do we provide predictions offline? Or do we do predictions on the fly? Dynamic has issues with latency, offline needs all the data available to make the prediction.

Data Dependencies


Be aware of features you add to your model. Questions to ask: 

Reliability - Is this data reliable and produced the same way?
Versioning - will this data change over time? Consider a version number
Necessity - does the usefulness of the feature justify the cost? Sometimes adding a feature is not worth it if the gains are minuscule.
Correlations - How are different features tied together?
Feedback loops - predictions of a model can affect the input. A stock market predictor can cause a stock to go up which causes the predictions to change which ...