So a friend just suggested a few minutes ago that i should remove most modifiers and just make the formula for success as d20 + more dice.
For skills it would pretty much work like D&D Next. He did gave an example for combat though that might be interesting.
Example: You have 1 rank in an attribute called Marksmanship, that attribute gives you a 1d4 to your attack rolls with ranged weapons. You are proficient with the longbow, so that gives you a +2 to your attack roll. Your opponent has 2 ranks in Agility that gives them 1d6 to checks with agility.
Your attack roll would be 1d20 + 1d4 + 2. Your opponent would roll to dodge with 1d20 + 1d6.
There is a lot of rolling involved and the math might slow the game down… i am just not sure.
Personally I think of flow in combat as a very important concept. I once played with a group of 4e players, and that daily spell the dwarf just got on his last level up took her as long to re-read as most other players spent on their entire turn. I mean, I am a power player, and there were times when I chose to go without a good spell simply because it was too complicated for combat.
Anyway, that rant was longer than it needed to be. But the point is that complexity =/= depth. On the other hand I do like this concept. But only if it being used for a purpose. You have to give thought to the fact that adding 1d4 is very different than adding any constant modifier. And each roll you add makes the odds more and more difficult to calculate. Finally I am worried that if this is a major source of modifiers you would end up having each attack roll require six or seven dice to be rolled and calculated, followed by rolling five dice for a defense score.
That being said, I am just trying to brainstorm here. If done properly it does sound like a fun concept. My suggestion would be to use the concept sparingly, and only to achieve a specific goal, or set of goals. Remember to keep those goals in mind when considering dice modifiers, and perhaps doing something to avoid them in both attack and defense rolls. Putting them on only attack rolls, or only on player characters?
Anyway. Those are my immediate thoughts on the concept. Good luck.
I can relate to your experience with 4e, both as a player and as a DM I tend to go for the simpler powers just to avoid slowing the game down (thanks for the video link btw).
Continuing with the concept, you wouldn’t roll more than one additional bonus die besides the d20. For example, a 3rd level character who has two ranks in marksmanship will have a d6 bonus to his rolls. Most abilities and feats will grant an increase the rank of the bonus die instead of adding an additional die. In a worse case scenario, you’ll be rolling 1d20 with two additional bonus dice.
There are still going to be solid modifiers, there just won’t be as many.

So a friend just suggested a few minutes ago that i should remove most modifiers and just make the formula for success as d20 + more dice.
For skills it would pretty much work like D&D Next. He did gave an example for combat though that might be interesting.
Example: You have 1 rank in an attribute called Marksmanship, that attribute gives you a 1d4 to your attack rolls with ranged weapons. You are proficient with the longbow, so that gives you a +2 to your attack roll. Your opponent has 2 ranks in Agility that gives them 1d6 to checks with agility.
Your attack roll would be 1d20 + 1d4 + 2. Your opponent would roll to dodge with 1d20 + 1d6.
There is a lot of rolling involved and the math might slow the game down… i am just not sure.
slashing, piercing, bludgeoning, fire, cold, thunder, necrotic, etc…

Some notes on the possible system for the game.
Tasks are resolved by 1d20 + skill + Misc Modifiers
Melee attacks are resolved by 1d20 + melee + weapon modifier
Melee damage is resolved by weapon die + melee
Range damage is resolved by 1d20 + Marksmanship + weapon modifier
Ranged damage is resolved by weapon die + marksmanship
Magic attacks are resolved by 1d20 + Willpower
Magic damage is resolved by spells’ damage + Willpower
Initiative: 1d20 + Agility + Athletics + Awareness
Dodge: 1d20 + Agility + Awareness
Parry: 1d20 + weapon modifier + Athletics (special rules for parrying attacks)
Players may get creative with ways they can use other skills to achieve something they could with another skill. For example, a character with a lumberjack background might have more athletics than melee training. While facing a werewolf, the player states that he should be able to use his athletics skill to make an attack with his axe instead of melee, he argues that his character is a lumberjack and should handle an axe pretty well. The player may have a point there, what you can do is allow that player to use 1/3 of the bonus granted by that skill (rounded down).
Another solution
Responses to tasks or attacks are resolved by making an opposed check either using the same skill when it comes to contests or attack vs a dodge or a parry.
Skills
Body
Mind
Spirit
Agility
Athletics
Marksmanship
Melee & Weaponry
Stealth
Thievery
Endurance
Academics
Arcana
Folklore
Forbidden Lore
Heraldry
Nature
Religion
Sciences
Warfare
Awareness
Deception
Handle Animal
Intimidation
Insight
Persuasion
Streetwise
Survival
Willpower
Vitality and wound Points
The vitality and wound points system was originally developed as a more cinematic method of handling damage than the traditional hit point system. The system allows for characters to improve the amount of punishment they can withstand as they go up in level, while still allowing for a single lucky attack to take down a character.
Metagame Analysis: Vitality And Wounds
Characters using this system should be more wary in combat, which can turn deadly in the space of a few lucky rolls. But they can also bounce back from a fight much more quickly. For that reason, this variant is an ideal system for low-magic campaigns or games where healing is otherwise rare.
A very weak creature in this system tends to be tougher to kill than in a standard d20 game. Very big creatures are also more durable, due to their size modifier.
Creatures capable of dealing a large amount of damage on a single hit become significantly more deadly in this system, since a lucky attack roll can give a deadly blow to almost any character.
Vitality Points
Vitality points are a measure of a character’s ability to turn a direct hit into a graze or a glancing blow with no serious consequences. Most types of damage reduce vitality points.
//VP recovers quickly outside of combat. You can use an action in combat to recover a bit.
For every point you have in Agility and Athletics, your vitality increases by 5 points. Your archetype gives you a bonus to your vitality points.
Wound Points
Wound points measure how much true physical damage a character can withstand. Damage reduces wound points only after all vitality points are gone, or when a character is struck by a critical hit.
You start with 10 wound points. For every point you have in Endurance, your wound increases by 5 points.
//Poison, some spells, and diseases tend to target WP directly. It recovers more slowly, generally only by resting.
A wraith is an undead creature born of evil and darkness, despising light and all living things. They can drain the life from living creatures, turning them into new wraiths upon death. Wraiths are powerless in natural sunlight, appearing as a sinister, spectral figure robed in darkness. They have no visual features or appendages, except for their glowing red eyes.